Oct 31 19:15:55.847274 ip-10-0-143-134 systemd[1]: Starting Kubernetes Kubelet... Oct 31 19:15:56.203811 ip-10-0-143-134 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:15:56.203811 ip-10-0-143-134 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 19:15:56.203811 ip-10-0-143-134 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:15:56.203811 ip-10-0-143-134 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:15:56.207823 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207717 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.207823 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207824 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207834 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207838 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207842 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207844 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207847 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207850 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207853 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207856 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207859 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207862 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207864 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207867 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207870 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207872 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207875 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207882 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207884 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207887 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207890 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.207902 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207893 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207896 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207898 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207901 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207903 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207906 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207909 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207912 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207915 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207918 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207921 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207924 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207926 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207929 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207932 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207934 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207937 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207939 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207942 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207944 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.208417 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207947 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207949 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207952 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207954 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207957 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207959 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207962 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207965 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207968 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207971 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207973 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207976 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207978 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207981 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207984 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207987 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.207990 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208005 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208009 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.208877 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208012 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208016 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208020 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208023 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208026 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208028 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208031 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208034 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208037 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208040 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208042 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208045 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208047 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208050 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208052 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208055 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208057 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208060 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208062 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.209369 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208065 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208067 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208071 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208073 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208076 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208078 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208081 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208083 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208088 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208091 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208093 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208096 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208101 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208105 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208110 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208114 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208118 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208121 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208125 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.209815 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208127 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208130 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208133 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208135 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208138 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208141 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208143 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.208146 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209525 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209533 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209536 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209539 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209542 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209545 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209547 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209550 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209553 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209556 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209559 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209562 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.210268 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209565 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209569 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209572 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209574 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209577 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209580 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209582 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209585 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209588 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209590 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209593 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209595 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209598 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209601 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209603 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209606 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209608 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209611 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209614 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.210750 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209616 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209618 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209621 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209624 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209626 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209629 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209632 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209634 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209637 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209639 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209642 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209645 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209647 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209650 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209653 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209657 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209659 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209662 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209664 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209667 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.211212 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209670 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209672 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209675 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209677 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209679 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209682 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209685 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209687 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209690 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209692 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209694 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209697 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209700 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209702 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209704 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209707 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209710 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209712 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209715 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209717 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.211708 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209720 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209722 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209725 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209730 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209732 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209736 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209739 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209742 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209746 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209749 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209752 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209755 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209758 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209761 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209763 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209766 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209768 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209772 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209777 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.212214 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209780 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209783 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209785 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209788 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209791 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209793 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209796 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209798 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209801 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209803 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209806 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209809 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209811 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209815 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209818 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.209820 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211091 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211098 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211103 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.212661 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211106 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211109 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211112 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211114 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211117 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211120 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211122 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211126 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211128 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211131 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211134 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211136 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211139 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211141 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211144 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211147 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211151 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211154 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211157 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211159 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.213114 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211162 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211165 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211167 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211170 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211172 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211175 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211178 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211180 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211183 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211185 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211189 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211192 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211195 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211197 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211201 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211205 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211208 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211211 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211213 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.213592 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211216 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211219 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211222 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211225 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211227 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211230 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211232 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211235 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211237 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211240 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211244 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211246 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211249 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211251 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211255 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211258 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211261 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211263 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211266 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.214058 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211268 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211271 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211273 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211276 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211278 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211282 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211284 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211287 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211289 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211292 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211294 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211297 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211300 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211302 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211304 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211307 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211310 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211312 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211314 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211317 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.214508 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211320 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211322 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211325 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211327 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211330 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211332 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211335 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211338 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211340 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211342 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211346 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211348 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211351 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211353 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211356 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211359 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211361 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211364 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211367 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.214964 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211369 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211372 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211375 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211377 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211380 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.211382 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211479 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211487 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211493 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211498 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211503 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211506 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211511 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211515 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211518 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211522 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211525 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211528 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211532 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211535 2575 flags.go:64] FLAG: --cgroup-root="" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211538 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211541 2575 flags.go:64] FLAG: --client-ca-file="" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211543 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 31 19:15:56.215420 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211546 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211551 2575 flags.go:64] FLAG: --cluster-domain="" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211554 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211557 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211560 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211564 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211568 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211571 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211574 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211578 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211581 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211584 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211587 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211590 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211593 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211598 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211601 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211604 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211606 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211609 2575 flags.go:64] FLAG: --enable-server="true" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211612 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211616 2575 flags.go:64] FLAG: --event-burst="100" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211619 2575 flags.go:64] FLAG: --event-qps="50" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211622 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 19:15:56.216037 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211626 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211629 2575 flags.go:64] FLAG: --eviction-hard="" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211633 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211636 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211639 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211642 2575 flags.go:64] FLAG: --eviction-soft="" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211645 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211648 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211651 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211654 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211657 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211660 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211663 2575 flags.go:64] FLAG: --feature-gates="" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211667 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211670 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211673 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211676 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211679 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211684 2575 flags.go:64] FLAG: --help="false" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211687 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211690 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211693 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211696 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 19:15:56.216644 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211699 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211703 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211705 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211708 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211711 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211714 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211718 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211723 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211726 2575 flags.go:64] FLAG: --kube-reserved="" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211730 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211733 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211736 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211739 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211742 2575 flags.go:64] FLAG: --lock-file="" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211745 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211748 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211752 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211758 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211760 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211764 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211766 2575 flags.go:64] FLAG: --logging-format="text" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211769 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211773 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211776 2575 flags.go:64] FLAG: --manifest-url="" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211779 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 31 19:15:56.217202 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211783 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211787 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211791 2575 flags.go:64] FLAG: --max-pods="110" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211794 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211798 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211801 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211804 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211807 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211810 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211813 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211819 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211822 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211825 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211836 2575 flags.go:64] FLAG: --pod-cidr="" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211840 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211843 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211848 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211852 2575 flags.go:64] FLAG: --port="10250" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211855 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211858 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f277ee16c1295c74" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211861 2575 flags.go:64] FLAG: --qos-reserved="" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211865 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211868 2575 flags.go:64] FLAG: --register-node="true" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211871 2575 flags.go:64] FLAG: --register-with-taints="" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211875 2575 flags.go:64] FLAG: --registry-burst="10" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211877 2575 flags.go:64] FLAG: --registry-qps="5" Oct 31 19:15:56.217830 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211880 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211883 2575 flags.go:64] FLAG: --reserved-memory="" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211887 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211890 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211893 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211896 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211899 2575 flags.go:64] FLAG: --runonce="false" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211902 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211905 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211908 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211912 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211915 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211918 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211921 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211924 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211928 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211931 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211933 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211936 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211939 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211944 2575 flags.go:64] FLAG: --system-cgroups="" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211947 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211954 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211957 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211960 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 19:15:56.218469 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211964 2575 flags.go:64] FLAG: --tls-min-version="" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211967 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211970 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211973 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211976 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211979 2575 flags.go:64] FLAG: --v="2" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211983 2575 flags.go:64] FLAG: --version="false" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.211987 2575 flags.go:64] FLAG: --vmodule="" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.212005 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.212010 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212142 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212146 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212149 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212152 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212155 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212158 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212160 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212163 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212166 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212169 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212171 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212174 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.219088 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212177 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212179 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212182 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212184 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212187 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212191 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212194 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212198 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212201 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212204 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212206 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212209 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212212 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212214 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212217 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212220 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212223 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212226 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212229 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212232 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.219590 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212235 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212237 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212240 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212243 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212245 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212248 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212250 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212253 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212256 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212259 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212261 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212264 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212267 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212269 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212272 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212274 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212277 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212281 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212283 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212288 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212290 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.220091 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212293 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212295 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212298 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212301 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212303 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212306 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212309 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212311 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212314 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212316 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212319 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212322 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212324 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212327 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212330 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212332 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212335 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212337 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212340 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.220609 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212343 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212346 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212349 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212351 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212354 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212357 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212359 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212362 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212364 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212368 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212371 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212375 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212378 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212382 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212386 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212389 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212391 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212394 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.221076 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212396 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212399 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212402 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212404 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212407 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212409 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212412 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212416 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212419 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212422 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212425 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212428 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212431 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212434 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212437 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.221497 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212439 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.212465 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212500 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212505 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212508 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212511 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212513 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212516 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212520 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212522 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212525 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212530 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212533 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212535 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212538 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212541 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.221855 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212543 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212546 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212549 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212551 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212554 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212557 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212560 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212562 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212565 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212567 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212570 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212572 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212575 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212578 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212580 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212583 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212585 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212588 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212591 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212594 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212597 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.222263 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212599 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212602 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212604 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212608 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212611 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212613 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212617 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212620 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212622 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212625 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212627 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212630 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212633 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212635 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212638 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212641 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212643 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212646 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212648 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.222770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212651 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212654 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212657 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212660 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212662 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212665 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212668 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212671 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212674 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212676 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212679 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212682 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212685 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212687 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212691 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212695 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212701 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212704 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.223317 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212707 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212710 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212713 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212716 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212718 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212721 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212724 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212726 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212730 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212733 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212736 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212739 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212742 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212744 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212747 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212749 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212752 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212754 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212757 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.223745 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212761 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212764 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212767 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212769 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212772 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212775 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212777 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212780 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212783 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212787 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212789 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212793 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212796 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212798 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.212802 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.218942 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.218959 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.218973 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.218978 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219045 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.224225 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219051 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219054 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219058 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219061 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219064 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219067 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219070 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219073 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219075 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219078 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219081 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219084 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219087 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219090 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219093 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219095 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219098 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219102 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219104 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219107 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.224706 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219110 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219114 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219117 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219121 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219124 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219126 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219129 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219132 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219136 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219139 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219142 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219145 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219147 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219150 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219153 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219156 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219159 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219161 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219164 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.225197 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219167 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219169 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219172 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219175 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219179 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219183 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219186 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219190 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219193 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219196 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219199 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219202 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219204 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219207 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219210 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219212 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219215 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219218 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219221 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.225770 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219223 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219226 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219229 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219232 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219235 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219238 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219240 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219243 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219245 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219248 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219251 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219253 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219256 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219258 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219261 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219263 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219266 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219268 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219271 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.226505 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219273 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219276 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219278 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219281 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219283 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219286 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219288 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219291 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219293 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219296 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219298 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219301 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219303 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219306 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219309 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219312 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219314 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219318 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219321 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219323 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.226952 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219326 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219328 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219331 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219333 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219336 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219339 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219341 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219343 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.219370 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219524 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219529 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219532 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219535 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219538 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219541 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.227646 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219543 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219546 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219549 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219551 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219554 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219557 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219559 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219562 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219564 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219568 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219571 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219574 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219578 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219580 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219583 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219586 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219589 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219592 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219594 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.228254 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219597 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219600 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219602 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219605 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219608 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219610 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219613 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219616 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219618 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219621 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219623 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219626 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219628 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219631 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219633 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219636 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219638 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219641 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219643 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.228813 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219646 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219648 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219651 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219653 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219656 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219658 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219661 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219663 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219666 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219669 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219672 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219674 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219677 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219679 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219682 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219685 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219689 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219692 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219694 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219697 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.229393 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219699 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219702 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219705 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219707 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219710 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219712 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219715 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219717 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219720 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219722 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219725 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219727 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219729 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219732 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219734 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219737 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219739 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219742 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219745 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219748 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.229907 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219751 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219753 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219756 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219759 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219761 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219764 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219766 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219769 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219771 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219774 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219776 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219779 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219781 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219784 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219790 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219792 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219795 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219797 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219800 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219802 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.230535 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219805 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219808 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.219838 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219871 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219875 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219878 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219880 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219883 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219885 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219888 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219891 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219893 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219896 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219899 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219901 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219904 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:15:56.231029 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219907 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219910 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219913 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219915 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219918 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219920 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219923 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219925 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219928 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219930 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219933 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219937 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219940 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219942 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219945 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219947 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219950 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219953 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219955 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:15:56.231404 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219958 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219960 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219963 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219965 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219968 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219970 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219974 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219977 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219980 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219982 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219985 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.219987 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220001 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220004 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220008 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220010 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220013 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220015 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220018 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220021 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:15:56.231848 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220023 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220026 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220028 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220031 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220035 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220037 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220040 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220043 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220045 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220048 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220051 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220054 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220056 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220059 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220062 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220064 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220067 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220070 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220072 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220075 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:15:56.232358 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220077 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220080 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220082 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220085 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220087 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220090 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220093 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220096 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220099 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220102 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220104 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220107 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220109 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220112 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220115 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220117 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220121 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220123 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220126 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220128 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:15:56.232894 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220131 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220133 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220136 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220138 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220141 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220144 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220146 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220148 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220153 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220157 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220161 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220163 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220166 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:56.220169 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.220227 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.221960 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.222700 2575 server.go:1014] "Starting client certificate rotation" Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.222790 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:15:56.233413 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.223388 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:15:56.233850 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.224444 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:15:56.233850 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.228348 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:15:56.237737 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.237724 2575 log.go:25] "Validated CRI v1 runtime API" Oct 31 19:15:56.240536 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.240522 2575 log.go:25] "Validated CRI v1 image API" Oct 31 19:15:56.241367 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.241340 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 19:15:56.243774 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.243753 2575 fs.go:135] Filesystem UUIDs: map[637e06d7-878b-4ba0-90a0-cd5e63e8eba5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c4a26a6c-9f1b-49b1-af84-246f59656ed8:/dev/nvme0n1p4] Oct 31 19:15:56.243840 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.243772 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 19:15:56.249888 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.249784 2575 manager.go:217] Machine: {Timestamp:2026-10-31 19:15:56.248503291 +0000 UTC m=+0.298274340 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099866 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8178473ea861f53639d68e37a0a3 SystemUUID:ec2b8178-473e-a861-f536-39d68e37a0a3 BootID:42918202-70f1-40da-8662-6cde8669caf8 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fb:65:9a:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fb:65:9a:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:7f:05:82:dd:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 19:15:56.249888 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.249887 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 19:15:56.249985 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.249965 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 19:15:56.252948 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.252922 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 19:15:56.253104 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.252949 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-134.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 19:15:56.253153 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.253108 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 19:15:56.253153 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.253116 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 19:15:56.253153 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.253152 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:15:56.253225 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.253170 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:15:56.253829 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.253819 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 19:15:56.257304 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.257292 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 19:15:56.257423 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.257414 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 19:15:56.258191 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.258182 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 31 19:15:56.258223 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.258196 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 19:15:56.258223 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.258206 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 19:15:56.258223 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.258216 2575 kubelet.go:405] "Adding apiserver pod source" Oct 31 19:15:56.258312 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.258224 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 19:15:56.258575 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.258563 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:15:56.260600 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.260574 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 19:15:56.264853 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.264841 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:15:56.265490 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265477 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 19:15:56.265522 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265508 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 19:15:56.265660 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265650 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 19:15:56.265689 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265665 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 19:15:56.265689 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265672 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 19:15:56.265689 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265678 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 19:15:56.265689 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265684 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 19:15:56.265689 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.265690 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 19:15:56.266325 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.266317 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 19:15:56.266325 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.266325 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 19:15:56.266385 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.266332 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 19:15:56.266385 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.266338 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 19:15:56.266385 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.266347 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 19:15:56.266385 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.266356 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 19:15:56.267502 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.267493 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 19:15:56.267502 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.267502 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 19:15:56.271791 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.271234 2575 server.go:1263] "Started kubelet" Oct 31 19:15:56.271791 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.271662 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 19:15:56.272119 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.272033 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 19:15:56.272205 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.272157 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 19:15:56.272621 ip-10-0-143-134 systemd[1]: Started Kubernetes Kubelet. Oct 31 19:15:56.274707 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.274689 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 19:15:56.274977 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.274958 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 31 19:15:56.277804 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.277783 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 19:15:56.279810 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.279798 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 19:15:56.279888 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.279869 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 19:15:56.280067 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.280023 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:56.280645 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280625 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 19:15:56.280741 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280693 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 19:15:56.280741 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280710 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 19:15:56.280839 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280829 2575 factory.go:153] Registering CRI-O factory Oct 31 19:15:56.280895 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280847 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 31 19:15:56.280895 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280855 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 31 19:15:56.280895 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280856 2575 factory.go:223] Registration of the crio container factory successfully Oct 31 19:15:56.281035 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280912 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 19:15:56.281035 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280924 2575 factory.go:55] Registering systemd factory Oct 31 19:15:56.281035 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280932 2575 factory.go:223] Registration of the systemd container factory successfully Oct 31 19:15:56.281035 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280947 2575 factory.go:103] Registering Raw factory Oct 31 19:15:56.281035 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.280962 2575 manager.go:1199] Started watching for new ooms in manager Oct 31 19:15:56.281720 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.281704 2575 manager.go:319] Starting recovery of all containers Oct 31 19:15:56.289033 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.289015 2575 manager.go:324] Recovery completed Oct 31 19:15:56.293335 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.293321 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.294811 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.294786 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.294874 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.294841 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.294874 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.294853 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.295799 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.295784 2575 cpu_manager.go:226] "Starting" policy="none" Oct 31 19:15:56.295799 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.295796 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 19:15:56.295903 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.295814 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 19:15:56.295903 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.295788 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.296542 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.296526 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.296589 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.296555 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.296589 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.296566 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.298039 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.298025 2575 policy_none.go:50] "Start" Oct 31 19:15:56.298102 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.298045 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 19:15:56.298102 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.298059 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 19:15:56.300684 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.300667 2575 policy_none.go:44] "Start" Oct 31 19:15:56.330087 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.330031 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 19:15:56.333328 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.333311 2575 manager.go:341] "Starting Device Plugin manager" Oct 31 19:15:56.333421 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.333346 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 19:15:56.333421 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.333356 2575 server.go:87] "Starting device plugin registration server" Oct 31 19:15:56.333739 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.333724 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 19:15:56.333814 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.333739 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 19:15:56.333858 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.333821 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 19:15:56.333932 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.333917 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 19:15:56.333932 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.333932 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 19:15:56.334502 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.334459 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 19:15:56.334604 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.334518 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:56.418476 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.417979 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-134.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 19:15:56.418476 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.417980 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-134.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 19:15:56.419092 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.417983 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.us-west-2.compute.internal.18e3b33e2e62f63e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:56.271171134 +0000 UTC m=+0.320942180,LastTimestamp:2026-10-31 19:15:56.271171134 +0000 UTC m=+0.320942180,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.us-west-2.compute.internal,}" Oct 31 19:15:56.422565 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.422535 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 19:15:56.423732 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.423711 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 19:15:56.423732 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.423735 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 19:15:56.423883 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.423757 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 19:15:56.423883 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.423793 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 19:15:56.434153 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.434135 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.434897 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.434884 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.434951 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.434912 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.434951 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.434924 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.434951 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.434948 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.524865 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.524775 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal"] Oct 31 19:15:56.524865 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.524848 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.526120 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.526102 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.526189 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.526132 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.526189 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.526145 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.528300 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.528289 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.528459 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.528446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.528505 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.528475 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.528943 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.528929 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.529039 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.528958 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.529039 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.528972 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.530978 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.530963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.531084 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531010 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.531084 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531072 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.531171 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531098 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.531171 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531109 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.531546 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531528 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 19:15:56.531652 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531607 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k7mp8" Oct 31 19:15:56.531696 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531655 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.531696 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531679 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.531696 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.531693 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.559462 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.559446 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.561843 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.561828 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.565249 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.565177 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.us-west-2.compute.internal.18e3b33e2fcba944 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-134.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:56.294809924 +0000 UTC m=+0.344580969,LastTimestamp:2026-10-31 19:15:56.294809924 +0000 UTC m=+0.344580969,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.us-west-2.compute.internal,}" Oct 31 19:15:56.565344 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.565271 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 19:15:56.565344 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.565271 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 19:15:56.565488 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.565468 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-134.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 19:15:56.681899 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.681869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.681899 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.681900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.682128 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.681989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76ead912f9e2b59f12d28159dae3cb89-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"76ead912f9e2b59f12d28159dae3cb89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.721315 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.721290 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:56.721638 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.721621 2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-134.us-west-2.compute.internal\" not found" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.774718 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.774694 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-134.us-west-2.compute.internal\" not found" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.782554 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.782499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.782554 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.782530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76ead912f9e2b59f12d28159dae3cb89-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"76ead912f9e2b59f12d28159dae3cb89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.782667 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.782555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.782667 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.782607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.782667 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.782608 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76ead912f9e2b59f12d28159dae3cb89-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"76ead912f9e2b59f12d28159dae3cb89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.782667 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.782635 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.862641 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.862613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.864757 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.864741 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.921809 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.921787 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:56.922744 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.922730 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:56.922803 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.922757 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:56.922803 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.922769 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:56.922803 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.922795 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.981395 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:56.981361 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:56.981395 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:56.981398 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-134.us-west-2.compute.internal\": node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.095582 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.095510 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.196542 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.196501 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.223286 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.223258 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 19:15:57.223697 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.223428 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 19:15:57.223697 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.223409 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa8143aa486024233b53a2014e030d47-5fe5f50e5295cc6b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-134.us-west-2.compute.internal.18e3b33e2fcc7539\": read tcp 10.0.143.134:38664->3.130.8.94:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-134.us-west-2.compute.internal.18e3b33e2fcc7539 default 5198 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-134.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 19:15:56 +0000 UTC,LastTimestamp:2026-10-31 19:15:56.434928628 +0000 UTC m=+0.484699673,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.us-west-2.compute.internal,}" Oct 31 19:15:57.297146 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.297113 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.398127 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.398063 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.498947 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.498915 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.536045 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.536011 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:10:56 +0000 UTC" deadline="2028-05-29 12:26:22.924957768 +0000 UTC" Oct 31 19:15:57.536149 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.536043 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13817h10m25.388923738s" Oct 31 19:15:57.599500 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.599473 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.602905 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.602889 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:15:57.603939 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.603922 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:15:57.604064 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.603956 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:15:57.604064 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.603973 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:15:57.684757 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.684673 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:57.700321 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.700299 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.800840 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.800802 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.901012 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:57.900965 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:15:57.974586 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.974526 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:57.980666 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:57.980644 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:58.038046 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.038014 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c6a40b90966d73cbc8b2f5cf79a1c2.slice/crio-53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383 WatchSource:0}: Error finding container 53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383: Status 404 returned error can't find the container with id 53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383 Oct 31 19:15:58.042731 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.042712 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:15:58.046950 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.046786 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:15:58.048023 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.048007 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:15:58.090469 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.090440 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ead912f9e2b59f12d28159dae3cb89.slice/crio-1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e WatchSource:0}: Error finding container 1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e: Status 404 returned error can't find the container with id 1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e Oct 31 19:15:58.111823 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.111799 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:15:58.195964 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.195932 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:58.260069 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.259962 2575 apiserver.go:52] "Watching apiserver" Oct 31 19:15:58.281978 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.281953 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 19:15:58.374159 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.374132 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 19:15:58.374476 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.374458 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-4s9m5","openshift-multus/network-metrics-daemon-kjnpm","openshift-network-diagnostics/network-check-target-lf79k","openshift-network-operator/iptables-alerter-76h2j","openshift-ovn-kubernetes/ovnkube-node-ngrxq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6","openshift-cluster-node-tuning-operator/tuned-p8qf2","openshift-dns/node-resolver-vs9x7","kube-system/global-pull-secret-syncer-gttls","openshift-multus/multus-2x85s","kube-system/konnectivity-agent-mxn7z","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal","openshift-image-registry/node-ca-dc5w2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal"] Oct 31 19:15:58.378352 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.378330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.380600 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.380571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.383055 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.383030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:15:58.383162 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.383112 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:15:58.385266 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.385247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.387793 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.387775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.390748 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.390730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.390850 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.390823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-conf\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.390850 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.390845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-lib-modules\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391155 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.390868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-systemd\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391228 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-var-lib-kubelet\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391288 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-tuned\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391288 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5aaedfd-3514-492d-96b2-2dc095ff2228-serviceca\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.391388 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8vc\" (UniqueName: \"kubernetes.io/projected/f5aaedfd-3514-492d-96b2-2dc095ff2228-kube-api-access-gl8vc\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.391388 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysconfig\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391388 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-run\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391388 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-sys\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391595 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5aaedfd-3514-492d-96b2-2dc095ff2228-host\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.391595 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-kubernetes\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391595 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-tmp\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391595 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66xx\" (UniqueName: \"kubernetes.io/projected/945c708d-ed9a-496a-8cfb-2531e8b9a15a-kube-api-access-f66xx\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391595 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:15:58.391833 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-host\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391833 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-modprobe-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.391833 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.391703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.394426 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.394408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.394505 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.394474 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:15:58.396860 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.396838 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.399439 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.399422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:15:58.399536 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.399489 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:15:58.401954 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.401932 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.404685 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.404663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.407135 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.407108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.428085 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.428038 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"76ead912f9e2b59f12d28159dae3cb89","Type":"ContainerStarted","Data":"1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e"} Oct 31 19:15:58.429099 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.429078 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"d6c6a40b90966d73cbc8b2f5cf79a1c2","Type":"ContainerStarted","Data":"53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383"} Oct 31 19:15:58.479170 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479115 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 19:15:58.479170 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479137 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.479382 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479256 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 19:15:58.479382 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479260 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.479382 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479303 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.479382 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479383 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 19:15:58.479690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479402 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.479690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479498 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.479690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479533 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.479690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479534 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 19:15:58.479690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479556 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.479690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479580 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.479938 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479692 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tqdrb\"" Oct 31 19:15:58.479938 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479776 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 19:15:58.479938 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.479862 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 19:15:58.480978 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.480876 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.480978 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.480912 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwj7k\"" Oct 31 19:15:58.480978 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.480922 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlrbg\"" Oct 31 19:15:58.481208 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.481060 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zsjk9\"" Oct 31 19:15:58.483873 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.483856 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 19:15:58.492369 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysconfig\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.492448 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.492448 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-multus-daemon-config\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.492448 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492434 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysconfig\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.492560 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzq4s\" (UniqueName: \"kubernetes.io/projected/e867aa15-55b5-48e1-8424-f5b176c3bcdb-kube-api-access-qzq4s\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.492560 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.492560 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2kr\" (UniqueName: \"kubernetes.io/projected/4fc92bb3-b6ec-4407-b9dd-6082981560e6-kube-api-access-sd2kr\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.492650 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-modprobe-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.492650 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.492714 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-cni-binary-copy\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.492714 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-hosts-file\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.492714 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-modprobe-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.492803 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e867aa15-55b5-48e1-8424-f5b176c3bcdb-iptables-alerter-script\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.492803 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.492803 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-system-cni-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.492897 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9976ed46-0a87-46f9-982d-4a4ac667e834-konnectivity-ca\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.492897 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492812 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.492897 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-node-log\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.492897 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-bin\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.493025 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-run\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493025 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-kubelet-config\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.493025 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-systemd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.493025 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.492985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-env-overrides\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.493172 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-etc-selinux\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.493172 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493043 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-run\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493172 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66xx\" (UniqueName: \"kubernetes.io/projected/945c708d-ed9a-496a-8cfb-2531e8b9a15a-kube-api-access-f66xx\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493172 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:15:58.493172 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-cnibin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.493320 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-kubelet\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.493320 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-hostroot\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.493320 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-systemd-units\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.493417 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-host\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493417 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-lib-modules\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493417 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-etc-kubernetes\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.493417 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-host\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493551 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493424 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-lib-modules\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493551 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-dbus\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.493551 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:15:58.493551 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-sys-fs\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.493551 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-systemd\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493551 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-var-lib-kubelet\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493582 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-var-lib-kubelet\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-systemd\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-tuned\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8vc\" (UniqueName: \"kubernetes.io/projected/f5aaedfd-3514-492d-96b2-2dc095ff2228-kube-api-access-gl8vc\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-conf-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9flg\" (UniqueName: \"kubernetes.io/projected/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-kube-api-access-x9flg\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-kubelet\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-var-lib-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-sys\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.493760 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493743 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-sys\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5aaedfd-3514-492d-96b2-2dc095ff2228-host\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-multus\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e867aa15-55b5-48e1-8424-f5b176c3bcdb-host-slash\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9976ed46-0a87-46f9-982d-4a4ac667e834-agent-certs\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493885 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5aaedfd-3514-492d-96b2-2dc095ff2228-host\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-config\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-kubernetes\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-tmp\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-socket-dir-parent\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-multus-certs\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.493998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494015 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-kubernetes\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58vt\" (UniqueName: \"kubernetes.io/projected/14c0ff06-8471-4aff-b1a2-ab2450b62cec-kube-api-access-m58vt\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-ovn\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-netd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-k8s-cni-cncf-io\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.494214 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494139 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-conf\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4rl\" (UniqueName: \"kubernetes.io/projected/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kube-api-access-zz4rl\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494254 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-conf\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-tmp-dir\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-socket-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rpc\" (UniqueName: \"kubernetes.io/projected/3c915558-1864-4e8a-8fdf-89439548044c-kube-api-access-86rpc\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-netns\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-script-lib\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-system-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-bin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zssg\" (UniqueName: \"kubernetes.io/projected/280701d1-1545-42b0-bb86-39832f46f516-kube-api-access-4zssg\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-slash\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.494773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-etc-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-log-socket\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-registration-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5aaedfd-3514-492d-96b2-2dc095ff2228-serviceca\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-netns\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cnibin\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.495324 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.494855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-device-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.495770 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.495746 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5aaedfd-3514-492d-96b2-2dc095ff2228-serviceca\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.497665 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.497643 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-tuned\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.497734 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.497685 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-tmp\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.541053 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.540980 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 19:15:58.541053 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.541036 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nwg6w" Oct 31 19:15:58.553547 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.553516 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:15:58.553547 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.553544 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:15:58.553744 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.553557 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6n2zr for pod openshift-network-diagnostics/network-check-target-lf79k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:58.553744 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.553659 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr podName:efc46de0-c9f5-4659-9713-631a24271f0d nodeName:}" failed. No retries permitted until 2026-10-31 19:15:59.053621152 +0000 UTC m=+3.103392203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6n2zr" (UniqueName: "kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr") pod "network-check-target-lf79k" (UID: "efc46de0-c9f5-4659-9713-631a24271f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:58.556018 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.555986 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8vc\" (UniqueName: \"kubernetes.io/projected/f5aaedfd-3514-492d-96b2-2dc095ff2228-kube-api-access-gl8vc\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.556115 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.555987 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66xx\" (UniqueName: \"kubernetes.io/projected/945c708d-ed9a-496a-8cfb-2531e8b9a15a-kube-api-access-f66xx\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.584705 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.584674 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.584705 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.584676 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.584705 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.584688 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 19:15:58.584985 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.584795 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 19:15:58.584985 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.584816 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4ftk8\"" Oct 31 19:15:58.584985 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.584954 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.584985 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.584979 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 19:15:58.585211 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585030 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 19:15:58.585211 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585095 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 19:15:58.585211 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585116 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 19:15:58.585211 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585131 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 19:15:58.585211 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585208 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 19:15:58.585396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585220 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 19:15:58.585396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585223 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2kswh\"" Oct 31 19:15:58.585396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585269 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vgfpk\"" Oct 31 19:15:58.585396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585271 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d7wnz\"" Oct 31 19:15:58.585396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.585338 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8492\"" Oct 31 19:15:58.595149 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4rl\" (UniqueName: \"kubernetes.io/projected/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kube-api-access-zz4rl\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.595230 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-tmp-dir\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.595230 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-socket-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.595336 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rpc\" (UniqueName: \"kubernetes.io/projected/3c915558-1864-4e8a-8fdf-89439548044c-kube-api-access-86rpc\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.595336 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-netns\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595336 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-script-lib\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-system-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.595478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-netns\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595357 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-socket-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.595478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-bin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.595478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595407 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-bin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.595478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.595478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zssg\" (UniqueName: \"kubernetes.io/projected/280701d1-1545-42b0-bb86-39832f46f516-kube-api-access-4zssg\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-slash\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-etc-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595559 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-slash\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-log-socket\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595610 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-etc-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-registration-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595657 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-log-socket\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-netns\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595695 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-tmp-dir\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595698 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-netns\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cnibin\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595731 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cnibin\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595743 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.595775 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595773 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-registration-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595801 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-device-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-system-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595822 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-device-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-multus-daemon-config\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzq4s\" (UniqueName: \"kubernetes.io/projected/e867aa15-55b5-48e1-8424-f5b176c3bcdb-kube-api-access-qzq4s\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2kr\" (UniqueName: \"kubernetes.io/projected/4fc92bb3-b6ec-4407-b9dd-6082981560e6-kube-api-access-sd2kr\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.595996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-cni-binary-copy\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-hosts-file\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e867aa15-55b5-48e1-8424-f5b176c3bcdb-iptables-alerter-script\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-system-cni-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.596593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596117 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596113 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-script-lib\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596178 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-hosts-file\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596116 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596198 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-system-cni-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9976ed46-0a87-46f9-982d-4a4ac667e834-konnectivity-ca\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-node-log\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-bin\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596407 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-node-log\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-kubelet-config\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596499 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-bin\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596524 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-multus-daemon-config\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-systemd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-env-overrides\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.596613 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-systemd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596696 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-cni-binary-copy\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.597399 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596718 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-kubelet-config\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-etc-selinux\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.596780 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret podName:fb981019-7d15-475e-ac02-ef33f915867a nodeName:}" failed. No retries permitted until 2026-10-31 19:15:59.096759452 +0000 UTC m=+3.146530502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret") pod "global-pull-secret-syncer-gttls" (UID: "fb981019-7d15-475e-ac02-ef33f915867a") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596814 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e867aa15-55b5-48e1-8424-f5b176c3bcdb-iptables-alerter-script\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-etc-selinux\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-cnibin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-kubelet\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596921 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-cnibin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596933 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-kubelet\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-hostroot\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596973 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-hostroot\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.596974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-systemd-units\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597019 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-etc-kubernetes\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597047 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-systemd-units\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-dbus\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597056 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-etc-kubernetes\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598126 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597102 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-dbus\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597108 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-sys-fs\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-conf-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.597159 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9flg\" (UniqueName: \"kubernetes.io/projected/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-kube-api-access-x9flg\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-kubelet\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:58.597213 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs podName:280701d1-1545-42b0-bb86-39832f46f516 nodeName:}" failed. No retries permitted until 2026-10-31 19:15:59.097197973 +0000 UTC m=+3.146969009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs") pod "network-metrics-daemon-kjnpm" (UID: "280701d1-1545-42b0-bb86-39832f46f516") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-sys-fs\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-conf-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-var-lib-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597248 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-kubelet\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-multus\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597276 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-var-lib-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e867aa15-55b5-48e1-8424-f5b176c3bcdb-host-slash\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597310 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-multus\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597354 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e867aa15-55b5-48e1-8424-f5b176c3bcdb-host-slash\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9976ed46-0a87-46f9-982d-4a4ac667e834-agent-certs\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.598610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-config\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-socket-dir-parent\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9976ed46-0a87-46f9-982d-4a4ac667e834-konnectivity-ca\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-multus-certs\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597523 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-env-overrides\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597533 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-socket-dir-parent\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597559 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-multus-certs\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58vt\" (UniqueName: \"kubernetes.io/projected/14c0ff06-8471-4aff-b1a2-ab2450b62cec-kube-api-access-m58vt\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597625 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-ovn\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-netd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-k8s-cni-cncf-io\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597717 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-ovn\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597765 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-k8s-cni-cncf-io\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-netd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597843 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599643 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597873 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599643 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.597916 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-config\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599643 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.598029 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.599643 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.598970 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.599643 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.599543 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9976ed46-0a87-46f9-982d-4a4ac667e834-agent-certs\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.659146 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.659123 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzq4s\" (UniqueName: \"kubernetes.io/projected/e867aa15-55b5-48e1-8424-f5b176c3bcdb-kube-api-access-qzq4s\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.659257 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.659144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4rl\" (UniqueName: \"kubernetes.io/projected/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kube-api-access-zz4rl\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.659257 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.659151 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rpc\" (UniqueName: \"kubernetes.io/projected/3c915558-1864-4e8a-8fdf-89439548044c-kube-api-access-86rpc\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.659257 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.659158 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zssg\" (UniqueName: \"kubernetes.io/projected/280701d1-1545-42b0-bb86-39832f46f516-kube-api-access-4zssg\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:15:58.659571 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.659554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2kr\" (UniqueName: \"kubernetes.io/projected/4fc92bb3-b6ec-4407-b9dd-6082981560e6-kube-api-access-sd2kr\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.659794 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.659779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58vt\" (UniqueName: \"kubernetes.io/projected/14c0ff06-8471-4aff-b1a2-ab2450b62cec-kube-api-access-m58vt\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.659831 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.659784 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9flg\" (UniqueName: \"kubernetes.io/projected/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-kube-api-access-x9flg\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.690582 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.690560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:15:58.698176 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.698152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:15:58.698596 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.698580 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:15:58.701123 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.701100 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945c708d_ed9a_496a_8cfb_2531e8b9a15a.slice/crio-57f77c0d8ea309db6dae1c3a483ea26c741952c51b3e7e4f82373c853d64add0 WatchSource:0}: Error finding container 57f77c0d8ea309db6dae1c3a483ea26c741952c51b3e7e4f82373c853d64add0: Status 404 returned error can't find the container with id 57f77c0d8ea309db6dae1c3a483ea26c741952c51b3e7e4f82373c853d64add0 Oct 31 19:15:58.706179 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.706160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:15:58.706756 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.706737 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5aaedfd_3514_492d_96b2_2dc095ff2228.slice/crio-253fd3cc9fdf1db01448e788d889783ed16d402e5807040136ddfe322a326763 WatchSource:0}: Error finding container 253fd3cc9fdf1db01448e788d889783ed16d402e5807040136ddfe322a326763: Status 404 returned error can't find the container with id 253fd3cc9fdf1db01448e788d889783ed16d402e5807040136ddfe322a326763 Oct 31 19:15:58.715169 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.715149 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode867aa15_55b5_48e1_8424_f5b176c3bcdb.slice/crio-4adbe3b5384c7b099b65ddd42f82dcc02935e7827d9710f01d79c48a17851017 WatchSource:0}: Error finding container 4adbe3b5384c7b099b65ddd42f82dcc02935e7827d9710f01d79c48a17851017: Status 404 returned error can't find the container with id 4adbe3b5384c7b099b65ddd42f82dcc02935e7827d9710f01d79c48a17851017 Oct 31 19:15:58.731931 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.731911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:15:58.738159 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.738140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2x85s" Oct 31 19:15:58.740402 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.740368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:15:58.742407 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.742387 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:15:58.742702 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.742676 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f90009c_b0e7_4288_ac5f_2a5fdc462b18.slice/crio-451293960e262de2f878c58e3f22f1ab4b7b2bf73b73841ecb19970de73be4f4 WatchSource:0}: Error finding container 451293960e262de2f878c58e3f22f1ab4b7b2bf73b73841ecb19970de73be4f4: Status 404 returned error can't find the container with id 451293960e262de2f878c58e3f22f1ab4b7b2bf73b73841ecb19970de73be4f4 Oct 31 19:15:58.747267 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.746680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:15:58.748758 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.748730 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c915558_1864_4e8a_8fdf_89439548044c.slice/crio-31909f6b80b3f5b5d87fd4ee5e839eec6d2d02837e0f124f826a955eeff82d27 WatchSource:0}: Error finding container 31909f6b80b3f5b5d87fd4ee5e839eec6d2d02837e0f124f826a955eeff82d27: Status 404 returned error can't find the container with id 31909f6b80b3f5b5d87fd4ee5e839eec6d2d02837e0f124f826a955eeff82d27 Oct 31 19:15:58.752375 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.752321 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc92bb3_b6ec_4407_b9dd_6082981560e6.slice/crio-9351ffa9e461c22db0c9cf0e5f2904897c0364879d7ab95694ff29d189ccbd78 WatchSource:0}: Error finding container 9351ffa9e461c22db0c9cf0e5f2904897c0364879d7ab95694ff29d189ccbd78: Status 404 returned error can't find the container with id 9351ffa9e461c22db0c9cf0e5f2904897c0364879d7ab95694ff29d189ccbd78 Oct 31 19:15:58.754798 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:58.754780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:15:58.755568 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.755547 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9976ed46_0a87_46f9_982d_4a4ac667e834.slice/crio-49f976ab4da4a06fc84b5b25b75e6423f07c857f74012908d5cade0b3602498b WatchSource:0}: Error finding container 49f976ab4da4a06fc84b5b25b75e6423f07c857f74012908d5cade0b3602498b: Status 404 returned error can't find the container with id 49f976ab4da4a06fc84b5b25b75e6423f07c857f74012908d5cade0b3602498b Oct 31 19:15:58.758342 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.758322 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d1a878_d2e1_4a6f_aa0b_ec6cd892b5a6.slice/crio-991eed3e6acf55afdbe8fc4215f40ee696edaae7e27b8fc657e3d400e3ed8ffc WatchSource:0}: Error finding container 991eed3e6acf55afdbe8fc4215f40ee696edaae7e27b8fc657e3d400e3ed8ffc: Status 404 returned error can't find the container with id 991eed3e6acf55afdbe8fc4215f40ee696edaae7e27b8fc657e3d400e3ed8ffc Oct 31 19:15:58.764258 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:15:58.764237 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c0ff06_8471_4aff_b1a2_ab2450b62cec.slice/crio-f0a7ad93fb0798da154a237136724b6a24d4d754ed38c3e7b6c5233f52fc30a3 WatchSource:0}: Error finding container f0a7ad93fb0798da154a237136724b6a24d4d754ed38c3e7b6c5233f52fc30a3: Status 404 returned error can't find the container with id f0a7ad93fb0798da154a237136724b6a24d4d754ed38c3e7b6c5233f52fc30a3 Oct 31 19:15:59.100871 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.100828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:15:59.101066 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.100881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:15:59.101066 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.100982 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:59.101066 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.101001 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:15:59.101066 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.101022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:15:59.101066 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.101037 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:15:59.101066 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.101050 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6n2zr for pod openshift-network-diagnostics/network-check-target-lf79k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:59.101066 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.101054 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret podName:fb981019-7d15-475e-ac02-ef33f915867a nodeName:}" failed. No retries permitted until 2026-10-31 19:16:00.101039003 +0000 UTC m=+4.150810041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret") pod "global-pull-secret-syncer-gttls" (UID: "fb981019-7d15-475e-ac02-ef33f915867a") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:15:59.101348 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.101092 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:59.101348 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.101100 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr podName:efc46de0-c9f5-4659-9713-631a24271f0d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:00.101084081 +0000 UTC m=+4.150855129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n2zr" (UniqueName: "kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr") pod "network-check-target-lf79k" (UID: "efc46de0-c9f5-4659-9713-631a24271f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:15:59.101348 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:15:59.101131 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs podName:280701d1-1545-42b0-bb86-39832f46f516 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:00.101120685 +0000 UTC m=+4.150891739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs") pod "network-metrics-daemon-kjnpm" (UID: "280701d1-1545-42b0-bb86-39832f46f516") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:15:59.433965 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.433866 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerStarted","Data":"f0a7ad93fb0798da154a237136724b6a24d4d754ed38c3e7b6c5233f52fc30a3"} Oct 31 19:15:59.435564 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.435532 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"9351ffa9e461c22db0c9cf0e5f2904897c0364879d7ab95694ff29d189ccbd78"} Oct 31 19:15:59.437702 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.437673 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2x85s" event={"ID":"3c915558-1864-4e8a-8fdf-89439548044c","Type":"ContainerStarted","Data":"31909f6b80b3f5b5d87fd4ee5e839eec6d2d02837e0f124f826a955eeff82d27"} Oct 31 19:15:59.439805 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.439776 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" event={"ID":"0f90009c-b0e7-4288-ac5f-2a5fdc462b18","Type":"ContainerStarted","Data":"451293960e262de2f878c58e3f22f1ab4b7b2bf73b73841ecb19970de73be4f4"} Oct 31 19:15:59.441431 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.441408 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-76h2j" event={"ID":"e867aa15-55b5-48e1-8424-f5b176c3bcdb","Type":"ContainerStarted","Data":"4adbe3b5384c7b099b65ddd42f82dcc02935e7827d9710f01d79c48a17851017"} Oct 31 19:15:59.443287 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.443261 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dc5w2" event={"ID":"f5aaedfd-3514-492d-96b2-2dc095ff2228","Type":"ContainerStarted","Data":"253fd3cc9fdf1db01448e788d889783ed16d402e5807040136ddfe322a326763"} Oct 31 19:15:59.445414 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.445387 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" event={"ID":"945c708d-ed9a-496a-8cfb-2531e8b9a15a","Type":"ContainerStarted","Data":"57f77c0d8ea309db6dae1c3a483ea26c741952c51b3e7e4f82373c853d64add0"} Oct 31 19:15:59.446946 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.446921 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vs9x7" event={"ID":"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6","Type":"ContainerStarted","Data":"991eed3e6acf55afdbe8fc4215f40ee696edaae7e27b8fc657e3d400e3ed8ffc"} Oct 31 19:15:59.448802 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.448776 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxn7z" event={"ID":"9976ed46-0a87-46f9-982d-4a4ac667e834","Type":"ContainerStarted","Data":"49f976ab4da4a06fc84b5b25b75e6423f07c857f74012908d5cade0b3602498b"} Oct 31 19:15:59.544702 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.544630 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:58 +0000 UTC" deadline="2028-08-04 00:19:04.192726062 +0000 UTC" Oct 31 19:15:59.544702 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:15:59.544694 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15413h3m4.648035936s" Oct 31 19:16:00.109136 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.109097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:00.109355 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.109152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:00.109355 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.109210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:00.109355 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109335 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:00.109524 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109400 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret podName:fb981019-7d15-475e-ac02-ef33f915867a nodeName:}" failed. No retries permitted until 2026-10-31 19:16:02.109381381 +0000 UTC m=+6.159152429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret") pod "global-pull-secret-syncer-gttls" (UID: "fb981019-7d15-475e-ac02-ef33f915867a") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:00.109597 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109493 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:00.109597 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109543 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:00.109597 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109558 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6n2zr for pod openshift-network-diagnostics/network-check-target-lf79k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:00.109750 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109604 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr podName:efc46de0-c9f5-4659-9713-631a24271f0d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:02.10958908 +0000 UTC m=+6.159360128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n2zr" (UniqueName: "kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr") pod "network-check-target-lf79k" (UID: "efc46de0-c9f5-4659-9713-631a24271f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:00.109809 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109776 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:00.109867 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.109828 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs podName:280701d1-1545-42b0-bb86-39832f46f516 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:02.109813931 +0000 UTC m=+6.159584984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs") pod "network-metrics-daemon-kjnpm" (UID: "280701d1-1545-42b0-bb86-39832f46f516") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:00.425277 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.425166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:00.425450 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.425307 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:00.425450 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.425420 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:00.425570 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.425519 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:00.428137 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.428110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:00.428269 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:00.428233 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:00.545285 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.545240 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:58 +0000 UTC" deadline="2028-08-14 13:21:28.889131576 +0000 UTC" Oct 31 19:16:00.545285 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:00.545283 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15666h5m28.343853098s" Oct 31 19:16:02.138678 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.138641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.138698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.138738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.138882 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.138946 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs podName:280701d1-1545-42b0-bb86-39832f46f516 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:06.138927084 +0000 UTC m=+10.188698130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs") pod "network-metrics-daemon-kjnpm" (UID: "280701d1-1545-42b0-bb86-39832f46f516") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.139017 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.139052 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret podName:fb981019-7d15-475e-ac02-ef33f915867a nodeName:}" failed. No retries permitted until 2026-10-31 19:16:06.139041164 +0000 UTC m=+10.188812210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret") pod "global-pull-secret-syncer-gttls" (UID: "fb981019-7d15-475e-ac02-ef33f915867a") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.139129 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.139147 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.139159 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6n2zr for pod openshift-network-diagnostics/network-check-target-lf79k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:02.139353 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.139189 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr podName:efc46de0-c9f5-4659-9713-631a24271f0d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:06.139180076 +0000 UTC m=+10.188951122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n2zr" (UniqueName: "kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr") pod "network-check-target-lf79k" (UID: "efc46de0-c9f5-4659-9713-631a24271f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:02.424436 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.424360 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:02.424583 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.424502 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:02.424641 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.424585 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:02.424691 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.424666 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:02.424729 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.424719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:02.424813 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:02.424788 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:02.496669 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.495580 2575 generic.go:356] "Generic (PLEG): container finished" podID="d6c6a40b90966d73cbc8b2f5cf79a1c2" containerID="f8eac77e9a830b3a6f89d152d4012c28f7def79a9c0bfe8d6e458db05123db1a" exitCode=0 Oct 31 19:16:02.496669 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:02.495636 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"d6c6a40b90966d73cbc8b2f5cf79a1c2","Type":"ContainerDied","Data":"f8eac77e9a830b3a6f89d152d4012c28f7def79a9c0bfe8d6e458db05123db1a"} Oct 31 19:16:03.511968 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:03.511723 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"76ead912f9e2b59f12d28159dae3cb89","Type":"ContainerStarted","Data":"3b1b6c64097eb583676478354f62dfd8816be480c0a1f74e362354a36af1270f"} Oct 31 19:16:03.519006 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:03.518962 2575 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="eef1ac0cb7cda0c0fc2f09f1341cb7c1f41cd000178cc855902fe663785bb394" exitCode=0 Oct 31 19:16:03.519134 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:03.519073 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerDied","Data":"eef1ac0cb7cda0c0fc2f09f1341cb7c1f41cd000178cc855902fe663785bb394"} Oct 31 19:16:03.528101 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:03.527342 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"d6c6a40b90966d73cbc8b2f5cf79a1c2","Type":"ContainerStarted","Data":"a9ec1b3f5409008362193202af4aed2675fce0d82b24b892a25cc6995eecb2b3"} Oct 31 19:16:03.626786 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:03.626346 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" podStartSLOduration=5.626326954 podStartE2EDuration="5.626326954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:03.625891848 +0000 UTC m=+7.675662904" watchObservedRunningTime="2026-10-31 19:16:03.626326954 +0000 UTC m=+7.676098009" Oct 31 19:16:03.862864 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:03.862812 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" podStartSLOduration=5.862794572 podStartE2EDuration="5.862794572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:03.741605603 +0000 UTC m=+7.791376662" watchObservedRunningTime="2026-10-31 19:16:03.862794572 +0000 UTC m=+7.912565627" Oct 31 19:16:04.436212 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:04.436180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:04.436382 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:04.436357 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:04.436480 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:04.436451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:04.436613 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:04.436590 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:04.436717 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:04.436683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:04.436866 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:04.436842 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:04.537289 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:04.537252 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxn7z" event={"ID":"9976ed46-0a87-46f9-982d-4a4ac667e834","Type":"ContainerStarted","Data":"eed1d1e5b598ae5d99c16b65f9c874e7db51bebd3d076f4d4c8acfb68410f035"} Oct 31 19:16:05.541660 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:05.541623 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" event={"ID":"0f90009c-b0e7-4288-ac5f-2a5fdc462b18","Type":"ContainerStarted","Data":"f7757660eb8e2884fc2a38c40b4c8bbcf638222c7a810dd275d61ed233cf2d06"} Oct 31 19:16:05.546646 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:05.546609 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dc5w2" event={"ID":"f5aaedfd-3514-492d-96b2-2dc095ff2228","Type":"ContainerStarted","Data":"4377b021458af3b5e3f9fce4bd135b8a0645918082feec44fd0f877f7f30b35d"} Oct 31 19:16:05.661944 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:05.661846 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mxn7z" podStartSLOduration=3.722898151 podStartE2EDuration="8.661824143s" totalImagesPullingTime="4.938925992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:04.654207895 +0000 UTC m=+8.703978952" watchObservedRunningTime="2026-10-31 19:16:05.661824143 +0000 UTC m=+9.711595199" Oct 31 19:16:06.170749 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:06.170715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:06.170908 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:06.170771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:06.170908 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:06.170802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:06.171048 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.170931 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:06.171048 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.170985 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs podName:280701d1-1545-42b0-bb86-39832f46f516 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.170968368 +0000 UTC m=+18.220739414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs") pod "network-metrics-daemon-kjnpm" (UID: "280701d1-1545-42b0-bb86-39832f46f516") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:06.171169 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.171091 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:06.171169 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.171132 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret podName:fb981019-7d15-475e-ac02-ef33f915867a nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.171119689 +0000 UTC m=+18.220890735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret") pod "global-pull-secret-syncer-gttls" (UID: "fb981019-7d15-475e-ac02-ef33f915867a") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:06.171263 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.171212 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:06.171263 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.171227 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:06.171263 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.171239 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6n2zr for pod openshift-network-diagnostics/network-check-target-lf79k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:06.171398 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.171273 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr podName:efc46de0-c9f5-4659-9713-631a24271f0d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:14.171262045 +0000 UTC m=+18.221033092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n2zr" (UniqueName: "kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr") pod "network-check-target-lf79k" (UID: "efc46de0-c9f5-4659-9713-631a24271f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:06.424307 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:06.424238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:06.424444 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.424370 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:06.424444 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:06.424438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:06.424534 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.424521 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:06.424572 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:06.424560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:06.424633 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:06.424616 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:07.549176 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:07.549083 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 19:16:07.557050 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:07.556841 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2x85s" event={"ID":"3c915558-1864-4e8a-8fdf-89439548044c","Type":"ContainerStarted","Data":"29242355430db2b8d0f89c45f019e791e4151d216a92dec9e3cc21c2bc3e652b"} Oct 31 19:16:07.560026 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:07.559985 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" event={"ID":"0f90009c-b0e7-4288-ac5f-2a5fdc462b18","Type":"ContainerStarted","Data":"db8a4be567e22e0fc4b5ef11b77ce9c35712b72c76d12500d0ebd8ea0772b9e5"} Oct 31 19:16:07.674889 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:07.674836 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dc5w2" podStartSLOduration=4.613219826 podStartE2EDuration="10.674817577s" totalImagesPullingTime="6.061597751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:05.663042746 +0000 UTC m=+9.712813801" watchObservedRunningTime="2026-10-31 19:16:07.674817577 +0000 UTC m=+11.724588632" Oct 31 19:16:08.340481 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.340263 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:16:07.549113864Z","UUID":"8e133596-ed2b-4453-8533-83b97f57254b","Handler":null,"Name":"","Endpoint":""} Oct 31 19:16:08.342011 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.341767 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 19:16:08.342011 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.341797 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 19:16:08.427127 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.427094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:08.427281 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:08.427194 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:08.427281 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.427265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:08.427389 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:08.427330 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:08.427389 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.427367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:08.427477 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:08.427418 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:08.504925 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.504340 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:16:08.505088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.504968 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:16:08.564655 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.564244 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vs9x7" event={"ID":"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6","Type":"ContainerStarted","Data":"9e070085d42213de97a948d8ac2f6982e9a168d4825a718df9c4113ed268ffee"} Oct 31 19:16:08.567170 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.567144 2575 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="93cd2a56b224b135da32a915e36b22ce5f4f328d77b6b95eb5e0c24301b2fa18" exitCode=0 Oct 31 19:16:08.568042 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.567986 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerDied","Data":"93cd2a56b224b135da32a915e36b22ce5f4f328d77b6b95eb5e0c24301b2fa18"} Oct 31 19:16:08.622270 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.621047 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2x85s" podStartSLOduration=3.232173876 podStartE2EDuration="11.621029195s" totalImagesPullingTime="8.388855319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:07.675218901 +0000 UTC m=+11.724989955" watchObservedRunningTime="2026-10-31 19:16:08.621029195 +0000 UTC m=+12.670800250" Oct 31 19:16:08.855338 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:08.855289 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vs9x7" podStartSLOduration=3.051849808 podStartE2EDuration="11.855276722s" totalImagesPullingTime="8.803426914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:08.854565956 +0000 UTC m=+12.904337010" watchObservedRunningTime="2026-10-31 19:16:08.855276722 +0000 UTC m=+12.905047776" Oct 31 19:16:09.572344 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:09.572165 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-76h2j" event={"ID":"e867aa15-55b5-48e1-8424-f5b176c3bcdb","Type":"ContainerStarted","Data":"5d208a39b3ee4fa7a7629cb4c196317f028028aa94eb5fe42b2f1c1aee38bce6"} Oct 31 19:16:09.574089 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:09.574064 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" event={"ID":"945c708d-ed9a-496a-8cfb-2531e8b9a15a","Type":"ContainerStarted","Data":"8b348c51f44553e2e6faeb6a5aadebf9dcfec71ee3169ed7e5faf60e9feb32b8"} Oct 31 19:16:09.578327 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:09.578305 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" event={"ID":"0f90009c-b0e7-4288-ac5f-2a5fdc462b18","Type":"ContainerStarted","Data":"a181bf036d04a58cf1b43b75eaa532f82bf629e0e2eda148753254a10de9f11c"} Oct 31 19:16:09.686086 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:09.686032 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-76h2j" podStartSLOduration=3.574714107 podStartE2EDuration="12.686014851s" totalImagesPullingTime="9.111300744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:09.685629652 +0000 UTC m=+13.735400709" watchObservedRunningTime="2026-10-31 19:16:09.686014851 +0000 UTC m=+13.735785906" Oct 31 19:16:09.802602 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:09.802551 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" podStartSLOduration=2.820933077 podStartE2EDuration="12.802536888s" totalImagesPullingTime="9.981603811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:09.802049892 +0000 UTC m=+13.851820947" watchObservedRunningTime="2026-10-31 19:16:09.802536888 +0000 UTC m=+13.852307941" Oct 31 19:16:09.920036 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:09.919963 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" podStartSLOduration=2.14647718 podStartE2EDuration="12.919943326s" totalImagesPullingTime="10.773466146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:09.919524512 +0000 UTC m=+13.969295566" watchObservedRunningTime="2026-10-31 19:16:09.919943326 +0000 UTC m=+13.969714381" Oct 31 19:16:10.424623 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:10.424584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:10.424810 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:10.424656 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:10.424810 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:10.424675 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:10.424810 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:10.424789 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:10.425018 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:10.424857 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:10.425018 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:10.424955 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:10.583641 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:10.583491 2575 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="66643323355a2a716f56a6fc4ed398a57873bbc60bab6c7eb7666603ae705e2c" exitCode=0 Oct 31 19:16:10.583641 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:10.583602 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerDied","Data":"66643323355a2a716f56a6fc4ed398a57873bbc60bab6c7eb7666603ae705e2c"} Oct 31 19:16:12.424435 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:12.424397 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:12.424435 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:12.424438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:12.425178 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:12.424408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:12.425178 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:12.424521 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:12.425178 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:12.424600 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:12.425178 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:12.424686 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:12.589842 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:12.589811 2575 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="e1c3a42bb23d25c948a2bc460361018cefa820035c8ab9f449b908e3995d87e8" exitCode=0 Oct 31 19:16:12.590013 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:12.589868 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerDied","Data":"e1c3a42bb23d25c948a2bc460361018cefa820035c8ab9f449b908e3995d87e8"} Oct 31 19:16:14.230378 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.230127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.230424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230266 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.230464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230496 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret podName:fb981019-7d15-475e-ac02-ef33f915867a nodeName:}" failed. No retries permitted until 2026-10-31 19:16:30.230480091 +0000 UTC m=+34.280251127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret") pod "global-pull-secret-syncer-gttls" (UID: "fb981019-7d15-475e-ac02-ef33f915867a") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230550 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230604 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs podName:280701d1-1545-42b0-bb86-39832f46f516 nodeName:}" failed. No retries permitted until 2026-10-31 19:16:30.23058808 +0000 UTC m=+34.280359129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs") pod "network-metrics-daemon-kjnpm" (UID: "280701d1-1545-42b0-bb86-39832f46f516") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230631 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230660 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230672 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6n2zr for pod openshift-network-diagnostics/network-check-target-lf79k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:14.230913 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.230713 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr podName:efc46de0-c9f5-4659-9713-631a24271f0d nodeName:}" failed. No retries permitted until 2026-10-31 19:16:30.230698828 +0000 UTC m=+34.280469879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n2zr" (UniqueName: "kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr") pod "network-check-target-lf79k" (UID: "efc46de0-c9f5-4659-9713-631a24271f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:14.424306 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.424272 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:14.424306 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.424294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:14.424528 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.424400 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:14.424528 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.424443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:14.424626 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.424539 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:14.424675 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:14.424618 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:14.596932 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.596725 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"1e51cc972134c037b238ba34c08da47d5778321504ec3b8888f114e348b684d1"} Oct 31 19:16:14.597086 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.596936 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"de801e36b54adf9fe056aa9b67dc2c2251f6caad7c60daa83d5fd45ae29a2154"} Oct 31 19:16:14.597086 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.596969 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"53e80e07fc286550f2d252d6bf7c81b7fc5ebecef3c92cf0380f6bc62b9e2919"} Oct 31 19:16:14.597086 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.597012 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"f303b3e607e9d6f56f5ec1b2ac23c0259830f78b599c05688cedcc9a8c96f9d3"} Oct 31 19:16:14.597086 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.597040 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"cab4e2d8aef07f9522cea965e80af040da3778d1441411dc02a1c8c42aa3e913"} Oct 31 19:16:14.597086 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:14.597066 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"00755c0eee71494f30691b50918948e0303ca40aa5216c85a01b839d8a5605cb"} Oct 31 19:16:15.601133 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:15.601102 2575 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="75ea1b57cbefd4d7c7ab2adaaac607c0a067c8983c852f2b8f644905a54182d7" exitCode=0 Oct 31 19:16:15.601563 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:15.601141 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerDied","Data":"75ea1b57cbefd4d7c7ab2adaaac607c0a067c8983c852f2b8f644905a54182d7"} Oct 31 19:16:16.425008 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:16.424962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:16.425163 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:16.425080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:16.425163 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:16.425096 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:16.425163 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:16.425143 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:16.425163 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:16.425157 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:16.425371 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:16.425226 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:16.605598 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:16.605512 2575 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="ce796ddf0b40d0922272a053f6615a9029d5672011dc9e687c026a4535bccc33" exitCode=0 Oct 31 19:16:16.606189 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:16.605588 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerDied","Data":"ce796ddf0b40d0922272a053f6615a9029d5672011dc9e687c026a4535bccc33"} Oct 31 19:16:16.608860 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:16.608841 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"587051e1f2de3c2660976fc6cf766238054d92e6e640e0055f6a61436e701a40"} Oct 31 19:16:17.614289 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:17.614255 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" event={"ID":"14c0ff06-8471-4aff-b1a2-ab2450b62cec","Type":"ContainerStarted","Data":"3f5e578ad9bcecb68da1eb521a751df5c357ac2ab958eda96ac902be7b302360"} Oct 31 19:16:17.834690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:17.834627 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4s9m5" podStartSLOduration=8.070726185 podStartE2EDuration="20.834612326s" totalImagesPullingTime="12.763886141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:17.783217195 +0000 UTC m=+21.832988249" watchObservedRunningTime="2026-10-31 19:16:17.834612326 +0000 UTC m=+21.884383380" Oct 31 19:16:18.424468 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:18.424431 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:18.424574 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:18.424437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:18.424702 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:18.424566 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:18.424702 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:18.424444 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:18.424702 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:18.424609 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:18.424852 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:18.424760 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:18.619329 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:18.619293 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" event={"ID":"4fc92bb3-b6ec-4407-b9dd-6082981560e6","Type":"ContainerStarted","Data":"e36aabc03b39d99401a6ec7f1faea6bda01831995f3da4b2ced9e687e91e35d6"} Oct 31 19:16:18.883361 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:18.883170 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" podStartSLOduration=6.941387228 podStartE2EDuration="21.883157987s" totalImagesPullingTime="14.941770759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:18.832311971 +0000 UTC m=+22.882083026" watchObservedRunningTime="2026-10-31 19:16:18.883157987 +0000 UTC m=+22.932929040" Oct 31 19:16:19.623854 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:19.623814 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:16:19.624421 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:19.624389 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:16:19.624563 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:19.624517 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:16:19.652495 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:19.652467 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:16:19.654514 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:19.654487 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:16:20.377400 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:20.377368 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:16:20.424789 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:20.424761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:20.424955 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:20.424824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:20.424955 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:20.424832 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:20.424955 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:20.424917 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:20.425118 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:20.425015 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:20.425153 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:20.425131 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:21.143901 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:21.143871 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gttls"] Oct 31 19:16:21.144344 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:21.143964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:21.144344 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:21.144057 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:21.146622 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:21.146602 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lf79k"] Oct 31 19:16:21.146721 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:21.146684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:21.146772 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:21.146758 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:21.147084 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:21.147069 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kjnpm"] Oct 31 19:16:21.147144 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:21.147134 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:21.147211 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:21.147198 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:22.127179 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:22.127117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxn7z_9976ed46-0a87-46f9-982d-4a4ac667e834/konnectivity-agent/0.log" limit=1024 Oct 31 19:16:22.424672 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:22.424586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:22.425164 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:22.424710 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:22.431398 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:22.431358 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxn7z_9976ed46-0a87-46f9-982d-4a4ac667e834/konnectivity-agent/0.log" limit=1024 Oct 31 19:16:22.729318 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:22.729283 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxn7z_9976ed46-0a87-46f9-982d-4a4ac667e834/konnectivity-agent/0.log" limit=1024 Oct 31 19:16:23.424300 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:23.424228 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:23.424300 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:23.424263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:23.424459 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:23.424336 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:23.424459 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:23.424449 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:23.529967 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:23.529943 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:16:23.530444 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:23.530060 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:16:23.530492 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:23.530470 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:16:24.424681 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:24.424651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:24.424838 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:24.424766 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:25.424161 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:25.424089 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:25.424161 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:25.424117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:25.424685 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:25.424204 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:25.424685 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:25.424279 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:25.885007 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:25.884968 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vs9x7_05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6/dns-node-resolver/0.log" Oct 31 19:16:26.419759 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:26.419731 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dc5w2_f5aaedfd-3514-492d-96b2-2dc095ff2228/node-ca/0.log" Oct 31 19:16:26.425612 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:26.425587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:26.426014 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:26.425706 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:27.424377 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:27.424345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:27.424543 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:27.424380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:27.424543 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:27.424447 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:27.424543 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:27.424508 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:28.424972 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:28.424943 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:28.425369 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:28.425062 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:29.424693 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:29.424654 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:29.424978 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:29.424701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:29.424978 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:29.424762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:29.424978 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:29.424903 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:30.243179 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:30.243133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:30.243199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:30.243227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243276 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243294 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243328 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243338 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs podName:280701d1-1545-42b0-bb86-39832f46f516 nodeName:}" failed. No retries permitted until 2026-10-31 19:17:02.24332422 +0000 UTC m=+66.293095252 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs") pod "network-metrics-daemon-kjnpm" (UID: "280701d1-1545-42b0-bb86-39832f46f516") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243345 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243352 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret podName:fb981019-7d15-475e-ac02-ef33f915867a nodeName:}" failed. No retries permitted until 2026-10-31 19:17:02.243346282 +0000 UTC m=+66.293117314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret") pod "global-pull-secret-syncer-gttls" (UID: "fb981019-7d15-475e-ac02-ef33f915867a") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:16:30.243361 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243354 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6n2zr for pod openshift-network-diagnostics/network-check-target-lf79k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:30.243630 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.243403 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr podName:efc46de0-c9f5-4659-9713-631a24271f0d nodeName:}" failed. No retries permitted until 2026-10-31 19:17:02.243389569 +0000 UTC m=+66.293160618 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6n2zr" (UniqueName: "kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr") pod "network-check-target-lf79k" (UID: "efc46de0-c9f5-4659-9713-631a24271f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:16:30.424364 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:30.424329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:30.424518 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:30.424465 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:31.424421 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:31.424390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:31.424836 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:31.424514 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:31.424836 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:31.424558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:31.424836 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:31.424626 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:32.424083 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:32.424053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:32.424242 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:32.424151 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:33.424562 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:33.424529 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:33.424938 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:33.424533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:33.424938 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:33.424634 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:33.424938 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:33.424711 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:34.424662 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:34.424627 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:34.425075 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:34.424745 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:35.424636 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:35.424600 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:35.424636 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:35.424630 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:35.425094 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:35.424702 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:35.425094 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:35.424813 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:36.424743 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:36.424711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:36.424939 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:36.424826 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:37.424045 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:37.424016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:37.424199 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:37.424017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:37.424199 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:37.424126 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf79k" podUID="efc46de0-c9f5-4659-9713-631a24271f0d" Oct 31 19:16:37.424199 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:37.424180 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gttls" podUID="fb981019-7d15-475e-ac02-ef33f915867a" Oct 31 19:16:38.424032 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:38.423989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:38.424394 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:38.424117 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kjnpm" podUID="280701d1-1545-42b0-bb86-39832f46f516" Oct 31 19:16:39.227687 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.227658 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeReady" Oct 31 19:16:39.227835 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.227773 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 19:16:39.343974 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.343942 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nlz6k"] Oct 31 19:16:39.345755 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.345739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.395552 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.395517 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dx4jp"] Oct 31 19:16:39.397435 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.397419 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2j7sr"] Oct 31 19:16:39.397594 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.397553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.399011 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.398976 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nlz6k"] Oct 31 19:16:39.399105 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.399087 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.400336 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.400317 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcs56\"" Oct 31 19:16:39.400439 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.400363 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 19:16:39.414529 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.414503 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dx4jp"] Oct 31 19:16:39.417833 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.417813 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2j7sr"] Oct 31 19:16:39.424944 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.424921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:16:39.425292 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.425136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:16:39.450167 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.450148 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 19:16:39.450247 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.450175 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 19:16:39.450735 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.450722 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 19:16:39.450802 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.450742 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 19:16:39.450802 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.450751 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 19:16:39.451864 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.451852 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 19:16:39.500938 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.500882 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\"" Oct 31 19:16:39.504434 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.501853 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgzhm\"" Oct 31 19:16:39.504434 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.501964 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:16:39.504434 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.502049 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4b89l\"" Oct 31 19:16:39.504434 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.502095 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 19:16:39.504434 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.502615 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\"" Oct 31 19:16:39.504434 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.502733 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 19:16:39.506546 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktr6\" (UniqueName: \"kubernetes.io/projected/e9968b83-2472-46f7-9ca4-664a3b8f89cd-kube-api-access-gktr6\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.506599 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84852974-1373-452c-b758-6a5e03bc5ac6-crio-socket\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.506599 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9968b83-2472-46f7-9ca4-664a3b8f89cd-metrics-tls\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.506700 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84852974-1373-452c-b758-6a5e03bc5ac6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.506747 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84852974-1373-452c-b758-6a5e03bc5ac6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.506747 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9968b83-2472-46f7-9ca4-664a3b8f89cd-tmp-dir\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.506863 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84852974-1373-452c-b758-6a5e03bc5ac6-data-volume\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.506902 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-cert\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.506902 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5rd\" (UniqueName: \"kubernetes.io/projected/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-kube-api-access-xs5rd\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.506970 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.506923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9968b83-2472-46f7-9ca4-664a3b8f89cd-config-volume\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.507028 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.507011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxr72\" (UniqueName: \"kubernetes.io/projected/84852974-1373-452c-b758-6a5e03bc5ac6-kube-api-access-mxr72\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.550815 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.550796 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:16:39.550972 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.550956 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:16:39.607412 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktr6\" (UniqueName: \"kubernetes.io/projected/e9968b83-2472-46f7-9ca4-664a3b8f89cd-kube-api-access-gktr6\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.607502 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84852974-1373-452c-b758-6a5e03bc5ac6-crio-socket\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.607502 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9968b83-2472-46f7-9ca4-664a3b8f89cd-metrics-tls\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.607596 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84852974-1373-452c-b758-6a5e03bc5ac6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.607596 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84852974-1373-452c-b758-6a5e03bc5ac6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.607596 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9968b83-2472-46f7-9ca4-664a3b8f89cd-tmp-dir\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.607596 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84852974-1373-452c-b758-6a5e03bc5ac6-data-volume\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.607773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607610 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84852974-1373-452c-b758-6a5e03bc5ac6-crio-socket\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.607911 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-cert\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.607964 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5rd\" (UniqueName: \"kubernetes.io/projected/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-kube-api-access-xs5rd\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.607964 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9968b83-2472-46f7-9ca4-664a3b8f89cd-config-volume\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.608085 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.607983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr72\" (UniqueName: \"kubernetes.io/projected/84852974-1373-452c-b758-6a5e03bc5ac6-kube-api-access-mxr72\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.608085 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.608044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84852974-1373-452c-b758-6a5e03bc5ac6-data-volume\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.608085 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.608045 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9968b83-2472-46f7-9ca4-664a3b8f89cd-tmp-dir\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.608296 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.608272 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84852974-1373-452c-b758-6a5e03bc5ac6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.608593 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.608571 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9968b83-2472-46f7-9ca4-664a3b8f89cd-config-volume\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.611199 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.611175 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9968b83-2472-46f7-9ca4-664a3b8f89cd-metrics-tls\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.611356 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.611336 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84852974-1373-452c-b758-6a5e03bc5ac6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.611448 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.611429 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-cert\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.665370 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.665343 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktr6\" (UniqueName: \"kubernetes.io/projected/e9968b83-2472-46f7-9ca4-664a3b8f89cd-kube-api-access-gktr6\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:39.665690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.665671 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5rd\" (UniqueName: \"kubernetes.io/projected/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-kube-api-access-xs5rd\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.666155 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.666135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr72\" (UniqueName: \"kubernetes.io/projected/84852974-1373-452c-b758-6a5e03bc5ac6-kube-api-access-mxr72\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.676059 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.676042 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:16:39.709959 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.709933 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:16:39.715104 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:39.715082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:40.046023 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.045981 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nlz6k"] Oct 31 19:16:40.046397 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:40.046373 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f4f0c2_7d69_487c_bfd0_cc4e5da89716.slice/crio-8ee3f2b1133b0df17678dd02b3a53b0ac45019a9bba2b8a5b70d45ab0a72b39a WatchSource:0}: Error finding container 8ee3f2b1133b0df17678dd02b3a53b0ac45019a9bba2b8a5b70d45ab0a72b39a: Status 404 returned error can't find the container with id 8ee3f2b1133b0df17678dd02b3a53b0ac45019a9bba2b8a5b70d45ab0a72b39a Oct 31 19:16:40.094302 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:40.094271 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9968b83_2472_46f7_9ca4_664a3b8f89cd.slice/crio-5865a98fd665c45f048f95b821958a0d8c60f51a41c92c3a051407a709acfd24 WatchSource:0}: Error finding container 5865a98fd665c45f048f95b821958a0d8c60f51a41c92c3a051407a709acfd24: Status 404 returned error can't find the container with id 5865a98fd665c45f048f95b821958a0d8c60f51a41c92c3a051407a709acfd24 Oct 31 19:16:40.096338 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.096311 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2j7sr"] Oct 31 19:16:40.096571 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:40.096552 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84852974_1373_452c_b758_6a5e03bc5ac6.slice/crio-f46f1c6d3380aa4ff141fa2d144bb00ebf76419c1459c6c272debffc3fa49b34 WatchSource:0}: Error finding container f46f1c6d3380aa4ff141fa2d144bb00ebf76419c1459c6c272debffc3fa49b34: Status 404 returned error can't find the container with id f46f1c6d3380aa4ff141fa2d144bb00ebf76419c1459c6c272debffc3fa49b34 Oct 31 19:16:40.098942 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.098923 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dx4jp"] Oct 31 19:16:40.425054 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.424976 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:16:40.478731 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.478698 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\"" Oct 31 19:16:40.478863 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.478742 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:16:40.666165 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.666130 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dx4jp" event={"ID":"84852974-1373-452c-b758-6a5e03bc5ac6","Type":"ContainerStarted","Data":"c852dad45379e1c1d4539825ec5cdf574790ea3134f8ee88c09c4d971d603690"} Oct 31 19:16:40.666317 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.666187 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dx4jp" event={"ID":"84852974-1373-452c-b758-6a5e03bc5ac6","Type":"ContainerStarted","Data":"f46f1c6d3380aa4ff141fa2d144bb00ebf76419c1459c6c272debffc3fa49b34"} Oct 31 19:16:40.667210 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.667186 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nlz6k" event={"ID":"61f4f0c2-7d69-487c-bfd0-cc4e5da89716","Type":"ContainerStarted","Data":"8ee3f2b1133b0df17678dd02b3a53b0ac45019a9bba2b8a5b70d45ab0a72b39a"} Oct 31 19:16:40.668147 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:40.668129 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j7sr" event={"ID":"e9968b83-2472-46f7-9ca4-664a3b8f89cd","Type":"ContainerStarted","Data":"5865a98fd665c45f048f95b821958a0d8c60f51a41c92c3a051407a709acfd24"} Oct 31 19:16:41.675033 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:41.674942 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dx4jp" event={"ID":"84852974-1373-452c-b758-6a5e03bc5ac6","Type":"ContainerStarted","Data":"caed59f034b5dd28d8f0244ec8d4e50fbf12c660803020d38be5498c5b11187d"} Oct 31 19:16:42.033657 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.033624 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vs9x7_05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6/dns-node-resolver/0.log" Oct 31 19:16:42.326893 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.326856 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4j7np"] Oct 31 19:16:42.328837 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.328820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.383584 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.383553 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4j7np"] Oct 31 19:16:42.429501 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.429466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.429659 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.429519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e993f09-bc04-46f4-9c3a-c77ba0341b81-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.429659 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.429603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.429750 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.429687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8pcl\" (UniqueName: \"kubernetes.io/projected/8e993f09-bc04-46f4-9c3a-c77ba0341b81-kube-api-access-f8pcl\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.433975 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.433942 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 19:16:42.433975 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.433980 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gxs8x\"" Oct 31 19:16:42.434285 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.434268 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 19:16:42.434353 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.434268 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 19:16:42.485414 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.485390 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 19:16:42.485562 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.485437 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 19:16:42.530879 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.530846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8pcl\" (UniqueName: \"kubernetes.io/projected/8e993f09-bc04-46f4-9c3a-c77ba0341b81-kube-api-access-f8pcl\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.530879 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.530892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.531128 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.530925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e993f09-bc04-46f4-9c3a-c77ba0341b81-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.531128 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.531094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.531668 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.531621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e993f09-bc04-46f4-9c3a-c77ba0341b81-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.533692 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.533673 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.533785 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.533754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.589124 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.589057 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8pcl\" (UniqueName: \"kubernetes.io/projected/8e993f09-bc04-46f4-9c3a-c77ba0341b81-kube-api-access-f8pcl\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.641093 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.641072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:16:42.833780 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:42.833761 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dc5w2_f5aaedfd-3514-492d-96b2-2dc095ff2228/node-ca/0.log" Oct 31 19:16:43.015253 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:43.015217 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e993f09_bc04_46f4_9c3a_c77ba0341b81.slice/crio-6620411e297a1abdfe5e246dcc8b49659358268e55b1a02ad2df65314458b27d WatchSource:0}: Error finding container 6620411e297a1abdfe5e246dcc8b49659358268e55b1a02ad2df65314458b27d: Status 404 returned error can't find the container with id 6620411e297a1abdfe5e246dcc8b49659358268e55b1a02ad2df65314458b27d Oct 31 19:16:43.016954 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:43.016855 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4j7np"] Oct 31 19:16:43.641037 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:43.640972 2575 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage366594280/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191642Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dae4e3a475c90e7014210d6b54e6810b8fd9db35f30dc2296401b11e1d18598d®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475102~hmac=7108a941dbeeb813b5b5562562ee4cb98b90fc6a71543eba82db1b3123ea00a7\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab" Oct 31 19:16:43.641253 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:43.641229 2575 kuberuntime_manager.go:1707] "Unhandled Error" err="container serve-healthcheck-canary start failed in pod ingress-canary-nlz6k_openshift-ingress-canary(61f4f0c2-7d69-487c-bfd0-cc4e5da89716): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage366594280/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191642Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dae4e3a475c90e7014210d6b54e6810b8fd9db35f30dc2296401b11e1d18598d®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475102~hmac=7108a941dbeeb813b5b5562562ee4cb98b90fc6a71543eba82db1b3123ea00a7\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError" Oct 31 19:16:43.641303 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:43.641274 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage366594280/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191642Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dae4e3a475c90e7014210d6b54e6810b8fd9db35f30dc2296401b11e1d18598d®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475102~hmac=7108a941dbeeb813b5b5562562ee4cb98b90fc6a71543eba82db1b3123ea00a7\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-nlz6k" podUID="61f4f0c2-7d69-487c-bfd0-cc4e5da89716" Oct 31 19:16:43.681705 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:43.681666 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j7sr" event={"ID":"e9968b83-2472-46f7-9ca4-664a3b8f89cd","Type":"ContainerStarted","Data":"f061e1b92accc29ebe4815abc490ba1d60b369a3debe8ed8cd431ec788fa260f"} Oct 31 19:16:43.681870 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:43.681727 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2j7sr" event={"ID":"e9968b83-2472-46f7-9ca4-664a3b8f89cd","Type":"ContainerStarted","Data":"8d08ad90cad34d9ab3776d494ed80bb1322a664f690ae70eae326af910b8f391"} Oct 31 19:16:43.681870 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:43.681808 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:43.682837 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:43.682810 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" event={"ID":"8e993f09-bc04-46f4-9c3a-c77ba0341b81","Type":"ContainerStarted","Data":"6620411e297a1abdfe5e246dcc8b49659358268e55b1a02ad2df65314458b27d"} Oct 31 19:16:43.684869 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:43.684846 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dx4jp" event={"ID":"84852974-1373-452c-b758-6a5e03bc5ac6","Type":"ContainerStarted","Data":"ad1c54f2238ce6a44a012e5ce34f00aa8b9d461ff7a7a67682622c2883b6d3f1"} Oct 31 19:16:43.686235 ip-10-0-143-134 kubenswrapper[2575]: E1031 19:16:43.686206 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage366594280/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T191642Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dae4e3a475c90e7014210d6b54e6810b8fd9db35f30dc2296401b11e1d18598d®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793475102~hmac=7108a941dbeeb813b5b5562562ee4cb98b90fc6a71543eba82db1b3123ea00a7\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-nlz6k" podUID="61f4f0c2-7d69-487c-bfd0-cc4e5da89716" Oct 31 19:16:43.800192 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:43.800141 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2j7sr" podStartSLOduration=2.273175633 podStartE2EDuration="4.800126657s" totalImagesPullingTime="2.526951024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:43.799063957 +0000 UTC m=+47.848835007" watchObservedRunningTime="2026-10-31 19:16:43.800126657 +0000 UTC m=+47.849897713" Oct 31 19:16:44.035690 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:44.035639 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dx4jp" podStartSLOduration=1.7947438390000001 podStartE2EDuration="5.035625834s" totalImagesPullingTime="3.240881995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:16:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:44.035183087 +0000 UTC m=+48.084954143" watchObservedRunningTime="2026-10-31 19:16:44.035625834 +0000 UTC m=+48.085396888" Oct 31 19:16:45.511952 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.509798 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f6fd84c5-t72cw"] Oct 31 19:16:45.512891 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.512871 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.568148 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.568118 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 19:16:45.619485 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.619456 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 19:16:45.619485 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.619456 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 19:16:45.619705 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.619692 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 19:16:45.619742 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.619697 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 19:16:45.619742 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.619697 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 19:16:45.619804 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.619745 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ptf8h\"" Oct 31 19:16:45.619870 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.619856 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 19:16:45.655903 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.655875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g88\" (UniqueName: \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.656019 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.655910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.656019 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.655941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.656099 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.656089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.656142 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.656115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.656142 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.656137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.656208 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.656152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.671893 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.671866 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f6fd84c5-t72cw"] Oct 31 19:16:45.691106 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.691077 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" event={"ID":"8e993f09-bc04-46f4-9c3a-c77ba0341b81","Type":"ContainerStarted","Data":"4c090eb5d5dd213b45f016df8d0868e0caa22914e778dd061a72630f696da9b5"} Oct 31 19:16:45.691222 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.691122 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" event={"ID":"8e993f09-bc04-46f4-9c3a-c77ba0341b81","Type":"ContainerStarted","Data":"ae768a04495e773eb659cd1773b25a75b471488325c77fda383e93d94956a950"} Oct 31 19:16:45.756914 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.756884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g88\" (UniqueName: \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.756914 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.756919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757117 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.756948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757181 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757245 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757245 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757245 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757559 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757527 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757806 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757785 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.757892 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.758012 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.757978 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.759540 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.759523 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.759603 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.759527 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.809140 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.809061 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" podStartSLOduration=1.938704188 podStartE2EDuration="3.809050397s" totalImagesPullingTime="1.870346209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:45.807740142 +0000 UTC m=+49.857511196" watchObservedRunningTime="2026-10-31 19:16:45.809050397 +0000 UTC m=+49.858821491" Oct 31 19:16:45.821365 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.821340 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g88\" (UniqueName: \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:45.823017 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:45.822980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:46.186588 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:46.186503 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f6fd84c5-t72cw"] Oct 31 19:16:46.213776 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:46.213751 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d70a7b_5d24_4d0f_909e_56b6b8273060.slice/crio-31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380 WatchSource:0}: Error finding container 31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380: Status 404 returned error can't find the container with id 31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380 Oct 31 19:16:46.694437 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:46.694397 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6fd84c5-t72cw" event={"ID":"d4d70a7b-5d24-4d0f-909e-56b6b8273060","Type":"ContainerStarted","Data":"31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380"} Oct 31 19:16:49.038626 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.038591 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gqmbj"] Oct 31 19:16:49.042256 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.042232 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180568 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180568 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-metrics-client-ca\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-textfile\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-wtmp\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180773 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-tls\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180935 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-root\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180935 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtmz\" (UniqueName: \"kubernetes.io/projected/73474282-1f87-4204-969d-2ddcbb5ee758-kube-api-access-vjtmz\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180935 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.180935 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.180923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-sys\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.212493 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.212466 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mvdsq\"" Oct 31 19:16:49.212661 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.212527 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 19:16:49.212661 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.212639 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 19:16:49.212759 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.212733 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 19:16:49.281519 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281519 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-metrics-client-ca\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281737 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-textfile\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281737 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-wtmp\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281737 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-tls\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281737 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-root\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281737 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtmz\" (UniqueName: \"kubernetes.io/projected/73474282-1f87-4204-969d-2ddcbb5ee758-kube-api-access-vjtmz\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281974 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281974 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-sys\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.281974 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.281899 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-sys\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.282308 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.282281 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-root\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.282665 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.282461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-metrics-client-ca\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.282665 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.282614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-wtmp\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.283159 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.282754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-textfile\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.283309 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.283285 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.284383 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.284357 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.284488 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.284400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-tls\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.341155 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.341091 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtmz\" (UniqueName: \"kubernetes.io/projected/73474282-1f87-4204-969d-2ddcbb5ee758-kube-api-access-vjtmz\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.360976 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.360948 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:16:49.378059 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:49.378028 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73474282_1f87_4204_969d_2ddcbb5ee758.slice/crio-aafd637df84a04707715c59df86cb8923ab7de1ca0d4ee0e3c02170a5b431bfc WatchSource:0}: Error finding container aafd637df84a04707715c59df86cb8923ab7de1ca0d4ee0e3c02170a5b431bfc: Status 404 returned error can't find the container with id aafd637df84a04707715c59df86cb8923ab7de1ca0d4ee0e3c02170a5b431bfc Oct 31 19:16:49.703302 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.703267 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6fd84c5-t72cw" event={"ID":"d4d70a7b-5d24-4d0f-909e-56b6b8273060","Type":"ContainerStarted","Data":"8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d"} Oct 31 19:16:49.704477 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.704455 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmbj" event={"ID":"73474282-1f87-4204-969d-2ddcbb5ee758","Type":"ContainerStarted","Data":"aafd637df84a04707715c59df86cb8923ab7de1ca0d4ee0e3c02170a5b431bfc"} Oct 31 19:16:49.821563 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:49.821514 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76f6fd84c5-t72cw" podStartSLOduration=1.481018302 podStartE2EDuration="4.821501225s" totalImagesPullingTime="3.340482923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:49.820892031 +0000 UTC m=+53.870663085" watchObservedRunningTime="2026-10-31 19:16:49.821501225 +0000 UTC m=+53.871272279" Oct 31 19:16:51.637984 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.637955 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:16:51.653414 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.653395 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-849d45575d-5ns87"] Oct 31 19:16:51.656506 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.656485 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.661498 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.661476 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-849d45575d-5ns87"] Oct 31 19:16:51.712070 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.712042 2575 generic.go:356] "Generic (PLEG): container finished" podID="73474282-1f87-4204-969d-2ddcbb5ee758" containerID="f5c6111f4c8739484cdf49dcb86ea94ec8c27d2b44b95eb838f8b8b5c12c6262" exitCode=0 Oct 31 19:16:51.712070 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.712072 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmbj" event={"ID":"73474282-1f87-4204-969d-2ddcbb5ee758","Type":"ContainerDied","Data":"f5c6111f4c8739484cdf49dcb86ea94ec8c27d2b44b95eb838f8b8b5c12c6262"} Oct 31 19:16:51.743478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.743396 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 19:16:51.743478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.743441 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 19:16:51.743478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.743447 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q82fc\"" Oct 31 19:16:51.743478 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.743462 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 19:16:51.743697 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.743500 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 19:16:51.743697 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.743530 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3sj0si341gcar\"" Oct 31 19:16:51.743697 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.743582 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 19:16:51.801861 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.801835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264ad1b2-15f9-4b71-97ef-96b7f380c650-metrics-client-ca\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.801971 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.801890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.802290 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.802256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.802388 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.802348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-grpc-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.802447 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.802384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt7g2\" (UniqueName: \"kubernetes.io/projected/264ad1b2-15f9-4b71-97ef-96b7f380c650-kube-api-access-tt7g2\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.802511 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.802464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.802511 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.802492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.802610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.802547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.802610 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.802581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/264ad1b2-15f9-4b71-97ef-96b7f380c650-tmp\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903697 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903697 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/264ad1b2-15f9-4b71-97ef-96b7f380c650-tmp\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903697 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264ad1b2-15f9-4b71-97ef-96b7f380c650-metrics-client-ca\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903962 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903962 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903962 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-grpc-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903962 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7g2\" (UniqueName: \"kubernetes.io/projected/264ad1b2-15f9-4b71-97ef-96b7f380c650-kube-api-access-tt7g2\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.903962 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.904235 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.903975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.904235 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.904137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/264ad1b2-15f9-4b71-97ef-96b7f380c650-tmp\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.904779 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.904747 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264ad1b2-15f9-4b71-97ef-96b7f380c650-metrics-client-ca\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.906495 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.906463 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.906633 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.906535 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.906801 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.906771 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.906871 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.906806 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.906919 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.906903 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.907128 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.907112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-grpc-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.962396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.962369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7g2\" (UniqueName: \"kubernetes.io/projected/264ad1b2-15f9-4b71-97ef-96b7f380c650-kube-api-access-tt7g2\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:51.979231 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:51.979211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:52.410704 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:52.410674 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264ad1b2_15f9_4b71_97ef_96b7f380c650.slice/crio-b0195ffa98da70adaf08c1f742bbbff32bb6e97f6837a23a1d9c22449a0fb379 WatchSource:0}: Error finding container b0195ffa98da70adaf08c1f742bbbff32bb6e97f6837a23a1d9c22449a0fb379: Status 404 returned error can't find the container with id b0195ffa98da70adaf08c1f742bbbff32bb6e97f6837a23a1d9c22449a0fb379 Oct 31 19:16:52.412892 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:52.412872 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-849d45575d-5ns87"] Oct 31 19:16:52.717545 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:52.717512 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmbj" event={"ID":"73474282-1f87-4204-969d-2ddcbb5ee758","Type":"ContainerStarted","Data":"be696f7fe66c7e83ad22a7f88c3f8be88dc2ca5ce3783ca3eccddffdd4b76baa"} Oct 31 19:16:52.718044 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:52.717570 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqmbj" event={"ID":"73474282-1f87-4204-969d-2ddcbb5ee758","Type":"ContainerStarted","Data":"a7e1cd73b6bd5bac8e1d544d68ddcc3ba95fa56855b29ced0ed00e41ee053290"} Oct 31 19:16:52.718721 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:52.718693 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" event={"ID":"264ad1b2-15f9-4b71-97ef-96b7f380c650","Type":"ContainerStarted","Data":"b0195ffa98da70adaf08c1f742bbbff32bb6e97f6837a23a1d9c22449a0fb379"} Oct 31 19:16:52.836962 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:52.836913 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gqmbj" podStartSLOduration=3.024657041 podStartE2EDuration="4.836900196s" totalImagesPullingTime="1.746104189s" totalInitContainerRuntime="66.138966ms" isStatefulPod=true podCreationTimestamp="2026-10-31 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:52.835802677 +0000 UTC m=+56.885573731" watchObservedRunningTime="2026-10-31 19:16:52.836900196 +0000 UTC m=+56.886671250" Oct 31 19:16:53.658493 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.658459 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-628gq"] Oct 31 19:16:53.661624 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.661603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:53.689797 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.689771 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2j7sr" Oct 31 19:16:53.713909 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.713885 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-628gq"] Oct 31 19:16:53.715786 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.715763 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 19:16:53.715916 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.715902 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5pcpp\"" Oct 31 19:16:53.820213 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.820177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:53.820692 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.820412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xg2\" (UniqueName: \"kubernetes.io/projected/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-kube-api-access-44xg2\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:53.823763 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.823736 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t"] Oct 31 19:16:53.826907 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.826883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:53.881524 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.881491 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 19:16:53.881656 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.881581 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 19:16:53.881656 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.881645 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 19:16:53.881748 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.881733 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bzkl5\"" Oct 31 19:16:53.881816 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.881803 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bqur2pghjms6i\"" Oct 31 19:16:53.881867 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.881856 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 19:16:53.882926 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.882903 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t"] Oct 31 19:16:53.921765 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.921687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:53.921765 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.921734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xg2\" (UniqueName: \"kubernetes.io/projected/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-kube-api-access-44xg2\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:53.924954 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.924924 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:53.980473 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:53.980442 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xg2\" (UniqueName: \"kubernetes.io/projected/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-kube-api-access-44xg2\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:54.022674 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.022602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-metrics-server-audit-profiles\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.022842 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.022697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-client-certs\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.022909 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.022864 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8n7\" (UniqueName: \"kubernetes.io/projected/df7cc259-0fb7-4f12-ac63-1b798ada6303-kube-api-access-7r8n7\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.022958 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.022936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-tls\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.023052 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.023034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-client-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.023110 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.023083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.023150 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.023120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df7cc259-0fb7-4f12-ac63-1b798ada6303-audit-log\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.123932 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.123898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-tls\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.124120 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.123952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-client-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.124184 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.124117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.124184 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.124179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df7cc259-0fb7-4f12-ac63-1b798ada6303-audit-log\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.124282 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.124214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-metrics-server-audit-profiles\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.124282 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.124244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-client-certs\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.124282 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.124264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8n7\" (UniqueName: \"kubernetes.io/projected/df7cc259-0fb7-4f12-ac63-1b798ada6303-kube-api-access-7r8n7\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.125656 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.125614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.125860 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.125836 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-metrics-server-audit-profiles\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.126097 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.126078 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df7cc259-0fb7-4f12-ac63-1b798ada6303-audit-log\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.127458 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.127433 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-client-certs\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.127704 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.127683 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-tls\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.128473 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.128453 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-client-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.182263 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.182207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8n7\" (UniqueName: \"kubernetes.io/projected/df7cc259-0fb7-4f12-ac63-1b798ada6303-kube-api-access-7r8n7\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.275554 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.275522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:54.439879 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.439641 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:16:54.640289 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:54.640261 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e5e0e1_8e0c_4224_bdea_f2a70ef90dcc.slice/crio-0687ff29840bcc8977d03289bcd53f416253d7ffa5c13bc787818c3cccce34e5 WatchSource:0}: Error finding container 0687ff29840bcc8977d03289bcd53f416253d7ffa5c13bc787818c3cccce34e5: Status 404 returned error can't find the container with id 0687ff29840bcc8977d03289bcd53f416253d7ffa5c13bc787818c3cccce34e5 Oct 31 19:16:54.642430 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.642409 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-628gq"] Oct 31 19:16:54.727075 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.727038 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" event={"ID":"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc","Type":"ContainerStarted","Data":"0687ff29840bcc8977d03289bcd53f416253d7ffa5c13bc787818c3cccce34e5"} Oct 31 19:16:54.804005 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:54.803957 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7cc259_0fb7_4f12_ac63_1b798ada6303.slice/crio-99b4ec3ec3b56b658df709725abf3a8191f2b0a15dd27d6eee2d3b5eec598024 WatchSource:0}: Error finding container 99b4ec3ec3b56b658df709725abf3a8191f2b0a15dd27d6eee2d3b5eec598024: Status 404 returned error can't find the container with id 99b4ec3ec3b56b658df709725abf3a8191f2b0a15dd27d6eee2d3b5eec598024 Oct 31 19:16:54.805140 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:54.805113 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t"] Oct 31 19:16:55.731162 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:55.731122 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" event={"ID":"df7cc259-0fb7-4f12-ac63-1b798ada6303","Type":"ContainerStarted","Data":"99b4ec3ec3b56b658df709725abf3a8191f2b0a15dd27d6eee2d3b5eec598024"} Oct 31 19:16:55.733305 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:55.733276 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" event={"ID":"264ad1b2-15f9-4b71-97ef-96b7f380c650","Type":"ContainerStarted","Data":"3e39cf5c7298cb93d8dd50f52f08acda921f3521f7152b26d57bf07e12b87e7c"} Oct 31 19:16:55.733396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:55.733329 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" event={"ID":"264ad1b2-15f9-4b71-97ef-96b7f380c650","Type":"ContainerStarted","Data":"91419d5dc1e00284e227b7b11d7401d0248012af9cfaf90afef4ac178bc6704c"} Oct 31 19:16:55.733396 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:55.733355 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" event={"ID":"264ad1b2-15f9-4b71-97ef-96b7f380c650","Type":"ContainerStarted","Data":"2bed9f9c095df6271687a698ba5e50fe4d04ce63d4360c62a550f6ac85894ec8"} Oct 31 19:16:55.823782 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:55.823750 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:55.823973 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:55.823808 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:55.829660 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:55.829636 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:56.745848 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:56.745820 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:16:57.659609 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.659575 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dcc66d7bc-g2fk7"] Oct 31 19:16:57.662694 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.662679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.748348 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.748313 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" event={"ID":"df7cc259-0fb7-4f12-ac63-1b798ada6303","Type":"ContainerStarted","Data":"d54c9b287e49c851dbf1b6a235e4135273e63d3b9ce2ba28392ba57e5d9eb276"} Oct 31 19:16:57.751789 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.751765 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" event={"ID":"264ad1b2-15f9-4b71-97ef-96b7f380c650","Type":"ContainerStarted","Data":"be7a688d3d1e3fd40e80953bad7ad10160006b81118377f190daa6f6b5720ed8"} Oct 31 19:16:57.751889 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.751826 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" event={"ID":"264ad1b2-15f9-4b71-97ef-96b7f380c650","Type":"ContainerStarted","Data":"0641582b3aa1058e1aacc032d3f646facab71636346cb20769645116187f2d4f"} Oct 31 19:16:57.751889 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.751859 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" event={"ID":"264ad1b2-15f9-4b71-97ef-96b7f380c650","Type":"ContainerStarted","Data":"3e633ac06206fb57f397a02ef65c6a0842275b777c8e57b42f89b7144025c2c9"} Oct 31 19:16:57.751988 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.751890 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:16:57.756176 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.754555 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" event={"ID":"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc","Type":"ContainerStarted","Data":"596fe158d107fc80269a15054766106266e673289e67c84d908e0d7a4394763a"} Oct 31 19:16:57.756176 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.754625 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:57.756454 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.756568 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.756568 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.756671 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.756671 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.756671 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.756834 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.756893 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.756847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhzf\" (UniqueName: \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.759926 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.759906 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:16:57.819681 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.819649 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 19:16:57.820809 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.820789 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcc66d7bc-g2fk7"] Oct 31 19:16:57.857379 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.857340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.857521 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.857452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.857839 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.857769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.857901 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.857856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhzf\" (UniqueName: \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.857963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.858010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.858232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.858410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.858634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.858745 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859088 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.859071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859550 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.859137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.859709 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.859691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.860651 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.860628 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.861462 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.861443 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.916615 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.916541 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhzf\" (UniqueName: \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:57.937857 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.937817 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" podStartSLOduration=2.622443736 podStartE2EDuration="4.937804076s" totalImagesPullingTime="2.31536034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:57.936915225 +0000 UTC m=+61.986686278" watchObservedRunningTime="2026-10-31 19:16:57.937804076 +0000 UTC m=+61.987575203" Oct 31 19:16:57.973391 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:57.973355 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:16:58.177216 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:58.177108 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" podStartSLOduration=3.083955929 podStartE2EDuration="7.17709583s" totalImagesPullingTime="4.093139901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:58.061194108 +0000 UTC m=+62.110965161" watchObservedRunningTime="2026-10-31 19:16:58.17709583 +0000 UTC m=+62.226866926" Oct 31 19:16:58.177501 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:58.177472 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" podStartSLOduration=3.098641697 podStartE2EDuration="5.177462972s" totalImagesPullingTime="2.078821275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:58.176359821 +0000 UTC m=+62.226130872" watchObservedRunningTime="2026-10-31 19:16:58.177462972 +0000 UTC m=+62.227234026" Oct 31 19:16:58.392702 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:16:58.392669 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931e6634_fb0c_498e_b225_59a8deb17655.slice/crio-283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529 WatchSource:0}: Error finding container 283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529: Status 404 returned error can't find the container with id 283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529 Oct 31 19:16:58.394832 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:58.394805 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcc66d7bc-g2fk7"] Oct 31 19:16:58.758050 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:58.758015 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc66d7bc-g2fk7" event={"ID":"931e6634-fb0c-498e-b225-59a8deb17655","Type":"ContainerStarted","Data":"021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997"} Oct 31 19:16:58.758430 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:58.758071 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc66d7bc-g2fk7" event={"ID":"931e6634-fb0c-498e-b225-59a8deb17655","Type":"ContainerStarted","Data":"283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529"} Oct 31 19:16:58.876096 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:16:58.876040 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5dcc66d7bc-g2fk7" podStartSLOduration=1.876026727 podStartE2EDuration="1.876026727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:58.874633985 +0000 UTC m=+62.924405038" watchObservedRunningTime="2026-10-31 19:16:58.876026727 +0000 UTC m=+62.925797781" Oct 31 19:17:02.314219 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.314133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:02.314219 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.314179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:02.314219 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.314208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:17:02.367192 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.367157 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:17:02.418858 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.418821 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:17:02.418858 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.418842 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:17:02.431197 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.427568 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:17:02.431544 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.431517 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:17:02.432484 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.432446 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:02.437679 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.437656 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:02.588457 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.588370 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\"" Oct 31 19:17:02.593339 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.593309 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\"" Oct 31 19:17:02.596376 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.596358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:02.601409 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.601390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:02.689967 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.689927 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\"" Oct 31 19:17:02.696238 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.696208 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:17:02.784761 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.784726 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nlz6k" event={"ID":"61f4f0c2-7d69-487c-bfd0-cc4e5da89716","Type":"ContainerStarted","Data":"efda1140303a44e6e0bdb42680a463a401b31c2e53ba2917485a59f7ba6ad1e9"} Oct 31 19:17:02.901337 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:02.901242 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nlz6k" podStartSLOduration=1.918467835 podStartE2EDuration="23.901227287s" totalImagesPullingTime="21.982759452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-31 19:17:02.90035908 +0000 UTC m=+66.950130134" watchObservedRunningTime="2026-10-31 19:17:02.901227287 +0000 UTC m=+66.950998341" Oct 31 19:17:03.048077 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:17:03.048046 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc46de0_c9f5_4659_9713_631a24271f0d.slice/crio-ecb3cc6249324da9b95c5d1db8b4388fe0047431a99d201ed00b719d79e4ced8 WatchSource:0}: Error finding container ecb3cc6249324da9b95c5d1db8b4388fe0047431a99d201ed00b719d79e4ced8: Status 404 returned error can't find the container with id ecb3cc6249324da9b95c5d1db8b4388fe0047431a99d201ed00b719d79e4ced8 Oct 31 19:17:03.049401 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:03.049377 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lf79k"] Oct 31 19:17:03.053380 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:03.053358 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gttls"] Oct 31 19:17:03.087323 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:17:03.087296 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb981019_7d15_475e_ac02_ef33f915867a.slice/crio-de47d30b150997640b6df36ad10c23589238c8a4ff1e529238fce29e26d78f5b WatchSource:0}: Error finding container de47d30b150997640b6df36ad10c23589238c8a4ff1e529238fce29e26d78f5b: Status 404 returned error can't find the container with id de47d30b150997640b6df36ad10c23589238c8a4ff1e529238fce29e26d78f5b Oct 31 19:17:03.098272 ip-10-0-143-134 kubenswrapper[2575]: W1031 19:17:03.098248 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280701d1_1545_42b0_bb86_39832f46f516.slice/crio-9658c19b90f33dc67e39586a7deb0982684d3416c176c8226b9126fa2e5e3c9a WatchSource:0}: Error finding container 9658c19b90f33dc67e39586a7deb0982684d3416c176c8226b9126fa2e5e3c9a: Status 404 returned error can't find the container with id 9658c19b90f33dc67e39586a7deb0982684d3416c176c8226b9126fa2e5e3c9a Oct 31 19:17:03.099917 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:03.099897 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kjnpm"] Oct 31 19:17:03.765177 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:03.765150 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:03.789290 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:03.789248 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kjnpm" event={"ID":"280701d1-1545-42b0-bb86-39832f46f516","Type":"ContainerStarted","Data":"9658c19b90f33dc67e39586a7deb0982684d3416c176c8226b9126fa2e5e3c9a"} Oct 31 19:17:03.790394 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:03.790373 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gttls" event={"ID":"fb981019-7d15-475e-ac02-ef33f915867a","Type":"ContainerStarted","Data":"de47d30b150997640b6df36ad10c23589238c8a4ff1e529238fce29e26d78f5b"} Oct 31 19:17:03.791531 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:03.791513 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lf79k" event={"ID":"efc46de0-c9f5-4659-9713-631a24271f0d","Type":"ContainerStarted","Data":"ecb3cc6249324da9b95c5d1db8b4388fe0047431a99d201ed00b719d79e4ced8"} Oct 31 19:17:05.804762 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:05.804721 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kjnpm" event={"ID":"280701d1-1545-42b0-bb86-39832f46f516","Type":"ContainerStarted","Data":"4453b62008f7b8bfc6fd9d17e18d0fc940a677ceac31c2a8295b785f60f88061"} Oct 31 19:17:05.805307 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:05.804784 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kjnpm" event={"ID":"280701d1-1545-42b0-bb86-39832f46f516","Type":"ContainerStarted","Data":"17d08ce682ffa1ff7b537c2c9f9cad6ad12751fddba6fd8f544c885fddb72c14"} Oct 31 19:17:05.925446 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:05.925386 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kjnpm" podStartSLOduration=67.264021868 podStartE2EDuration="1m8.925369861s" totalImagesPullingTime="1.661347993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:05.923852402 +0000 UTC m=+69.973623457" watchObservedRunningTime="2026-10-31 19:17:05.925369861 +0000 UTC m=+69.975140915" Oct 31 19:17:07.815538 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:07.814630 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lf79k" event={"ID":"efc46de0-c9f5-4659-9713-631a24271f0d","Type":"ContainerStarted","Data":"4891938d4e2c5f13511c3048c046168345f2a29775f8ddf4a4144244db5c1c0e"} Oct 31 19:17:07.815538 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:07.815493 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:07.936901 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:07.936848 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lf79k" podStartSLOduration=66.944285595 podStartE2EDuration="1m10.936831356s" totalImagesPullingTime="3.992545761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:07.936487742 +0000 UTC m=+71.986258799" watchObservedRunningTime="2026-10-31 19:17:07.936831356 +0000 UTC m=+71.986602412" Oct 31 19:17:07.973935 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:07.973900 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:07.974124 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:07.973963 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:07.979369 ip-10-0-143-134 kubenswrapper[2575]: I1031 19:17:07.979343 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:08.706342 ip-10-0-143-134 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 19:17:08.721273 ip-10-0-143-134 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 19:17:08.721434 ip-10-0-143-134 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 19:17:08.721491 ip-10-0-143-134 systemd[1]: kubelet.service: Consumed 3.950s CPU time. Oct 31 19:17:08.728619 ip-10-0-143-134 systemd[1]: Starting Kubernetes Kubelet... Oct 31 19:17:08.789006 ip-10-0-143-134 kubenswrapper[6481]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:17:08.789006 ip-10-0-143-134 kubenswrapper[6481]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 19:17:08.789006 ip-10-0-143-134 kubenswrapper[6481]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:17:08.789006 ip-10-0-143-134 kubenswrapper[6481]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:17:08.791338 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791255 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.791338 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791333 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.791338 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791339 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791346 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791351 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791355 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791360 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791364 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791368 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791372 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791376 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791380 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791384 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791388 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791393 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791404 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791408 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791412 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791417 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791421 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791425 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.791551 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791429 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791433 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791438 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791441 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791445 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791449 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791453 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791457 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791462 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791469 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791476 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791481 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791487 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791492 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791496 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791501 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791505 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791509 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791514 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.792386 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791520 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791525 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791529 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791533 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791537 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791542 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791546 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791549 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791554 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791559 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791563 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791568 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791572 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791576 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791580 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791585 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791589 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791593 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791598 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791604 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.793236 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791610 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791614 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791618 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791622 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791627 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791631 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791635 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791640 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791644 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791648 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791653 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791657 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791664 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791669 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791675 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791679 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791684 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791688 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791692 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.794095 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791696 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791700 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791707 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791711 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791716 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791721 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791725 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791729 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791734 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791738 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791742 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791746 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791751 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791755 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791760 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791764 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791768 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791773 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791777 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.794940 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791782 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791786 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791791 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791797 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791801 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791806 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791811 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.791815 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793074 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793085 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793090 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793095 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793100 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793104 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793108 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793112 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793118 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793122 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793126 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793130 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.795814 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793135 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793139 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793143 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793147 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793154 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793162 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793167 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793172 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793176 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793180 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793184 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793188 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793192 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793197 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793201 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793206 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793210 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793214 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793219 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.796707 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793223 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793227 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793231 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793235 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793239 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793243 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793248 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793252 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793256 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793260 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793265 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793270 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793274 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793279 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793283 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793289 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793295 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793301 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793306 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.797566 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793312 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793317 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793321 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793325 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793330 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793334 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793338 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793343 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793347 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793351 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793355 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793360 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793364 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793368 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793372 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793376 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793380 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793384 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793389 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793393 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.798097 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793397 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793401 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793405 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793410 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793415 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793419 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793423 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793428 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793433 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793438 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793446 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793451 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793455 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793459 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793464 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793468 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793472 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793477 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793481 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.798627 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793486 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793491 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793495 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793499 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793503 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793510 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793515 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793519 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793523 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793528 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793532 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793536 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793540 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793544 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793549 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793553 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.793557 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794366 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794376 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794381 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.799103 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794385 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794390 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794395 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794402 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794407 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794411 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794415 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794420 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794424 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794429 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794433 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794438 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794442 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794447 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794452 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794457 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794461 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794465 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794470 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.799630 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794474 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794479 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794483 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794487 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794491 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794496 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794500 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794504 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794508 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794513 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794517 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794522 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794527 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794532 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794536 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794540 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794546 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794551 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794556 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.800136 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794561 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794565 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794569 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794574 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794578 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794582 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794586 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794591 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794595 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794599 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794604 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794608 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794613 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794617 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794621 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794625 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794629 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794634 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794638 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794644 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.800586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794648 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794653 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794657 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794661 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794666 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794671 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794676 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794691 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794696 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794703 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794709 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794713 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794717 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794722 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794726 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794730 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794734 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794738 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794743 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794747 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.801271 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794751 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794755 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794760 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794764 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794768 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794773 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794779 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794787 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794791 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794796 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794801 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794805 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794809 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794813 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794817 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794822 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794826 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794830 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794835 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.801919 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794841 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794845 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794854 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794858 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794862 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.794866 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795019 6481 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795030 6481 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795039 6481 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795047 6481 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795053 6481 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795058 6481 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795065 6481 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795071 6481 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795077 6481 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795082 6481 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795088 6481 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795093 6481 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795098 6481 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795103 6481 flags.go:64] FLAG: --cgroup-root="" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795108 6481 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795113 6481 flags.go:64] FLAG: --client-ca-file="" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795118 6481 flags.go:64] FLAG: --cloud-provider="external" Oct 31 19:17:08.802406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795123 6481 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795130 6481 flags.go:64] FLAG: --cluster-domain="" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795135 6481 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795140 6481 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795145 6481 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795150 6481 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795157 6481 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795162 6481 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795166 6481 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795183 6481 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795188 6481 flags.go:64] FLAG: --contention-profiling="false" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795197 6481 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795203 6481 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795208 6481 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795212 6481 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795225 6481 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795230 6481 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795235 6481 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795240 6481 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795245 6481 flags.go:64] FLAG: --enable-server="true" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795249 6481 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795256 6481 flags.go:64] FLAG: --event-burst="100" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795261 6481 flags.go:64] FLAG: --event-qps="50" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795266 6481 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 19:17:08.802969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795271 6481 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795275 6481 flags.go:64] FLAG: --eviction-hard="" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795282 6481 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795287 6481 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795293 6481 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795298 6481 flags.go:64] FLAG: --eviction-soft="" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795302 6481 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795307 6481 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795312 6481 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795316 6481 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795320 6481 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795325 6481 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795330 6481 flags.go:64] FLAG: --feature-gates="" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795337 6481 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795342 6481 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795347 6481 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795352 6481 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795357 6481 flags.go:64] FLAG: --healthz-port="10248" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795363 6481 flags.go:64] FLAG: --help="false" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795370 6481 flags.go:64] FLAG: --hostname-override="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795375 6481 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795380 6481 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795385 6481 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 19:17:08.803688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795391 6481 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795396 6481 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795401 6481 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795405 6481 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795410 6481 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795415 6481 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795419 6481 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795424 6481 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795429 6481 flags.go:64] FLAG: --kube-reserved="" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795434 6481 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795439 6481 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795444 6481 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795449 6481 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795455 6481 flags.go:64] FLAG: --lock-file="" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795459 6481 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795464 6481 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795469 6481 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795478 6481 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795482 6481 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795487 6481 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795492 6481 flags.go:64] FLAG: --logging-format="text" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795496 6481 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795502 6481 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795506 6481 flags.go:64] FLAG: --manifest-url="" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795511 6481 flags.go:64] FLAG: --manifest-url-header="" Oct 31 19:17:08.804252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795518 6481 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795523 6481 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795529 6481 flags.go:64] FLAG: --max-pods="110" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795537 6481 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795542 6481 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795547 6481 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795551 6481 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795556 6481 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795561 6481 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795566 6481 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795576 6481 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795581 6481 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795585 6481 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795590 6481 flags.go:64] FLAG: --pod-cidr="" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795595 6481 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795600 6481 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795605 6481 flags.go:64] FLAG: --pods-per-core="0" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795609 6481 flags.go:64] FLAG: --port="10250" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795614 6481 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795619 6481 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f277ee16c1295c74" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795624 6481 flags.go:64] FLAG: --qos-reserved="" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795629 6481 flags.go:64] FLAG: --read-only-port="10255" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795633 6481 flags.go:64] FLAG: --register-node="true" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795638 6481 flags.go:64] FLAG: --register-with-taints="" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795645 6481 flags.go:64] FLAG: --registry-burst="10" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795649 6481 flags.go:64] FLAG: --registry-qps="5" Oct 31 19:17:08.804838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795654 6481 flags.go:64] FLAG: --reserved-cpus="" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795658 6481 flags.go:64] FLAG: --reserved-memory="" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795664 6481 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795669 6481 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795673 6481 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795678 6481 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795683 6481 flags.go:64] FLAG: --runonce="false" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795688 6481 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795693 6481 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795700 6481 flags.go:64] FLAG: --seccomp-default="false" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795705 6481 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795710 6481 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795715 6481 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795720 6481 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795725 6481 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795730 6481 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795735 6481 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795740 6481 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795745 6481 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795750 6481 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795756 6481 flags.go:64] FLAG: --system-cgroups="" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795761 6481 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795769 6481 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795774 6481 flags.go:64] FLAG: --tls-cert-file="" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795778 6481 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 19:17:08.805514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795785 6481 flags.go:64] FLAG: --tls-min-version="" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795790 6481 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795794 6481 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795799 6481 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795804 6481 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795809 6481 flags.go:64] FLAG: --v="2" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795815 6481 flags.go:64] FLAG: --version="false" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795821 6481 flags.go:64] FLAG: --vmodule="" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795827 6481 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.795832 6481 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796058 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796066 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796070 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796075 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796080 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796084 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796091 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796095 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796101 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796105 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796110 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.806142 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796114 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796121 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796128 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796133 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796139 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796143 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796149 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796153 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796157 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796162 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796166 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796171 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796175 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796180 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796184 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796188 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796192 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796196 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796200 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.806637 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796205 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796210 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796214 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796218 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796222 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796226 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796231 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796235 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796241 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796245 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796250 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796255 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796260 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796264 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796268 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796272 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796276 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796281 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796285 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796291 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.807206 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796295 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796300 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796304 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796308 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796313 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796317 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796321 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796326 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796330 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796334 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796339 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796343 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796347 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796351 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796355 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796360 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796365 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796369 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796374 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796378 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.807787 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796385 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796390 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796394 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796398 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796403 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796407 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796412 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796419 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796425 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796429 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796435 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796441 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796446 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796450 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796455 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796459 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796464 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796468 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796473 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796477 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.808312 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796481 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796485 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796489 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796494 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796498 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796502 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796506 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796511 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796515 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796519 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796524 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796528 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796534 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796538 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796542 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796546 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.808807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.796589 6481 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796644 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796652 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796657 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796661 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796665 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796669 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796674 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796681 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796685 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796690 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796694 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796698 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796702 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796706 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796711 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796714 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796718 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796722 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796727 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.809240 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796731 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796735 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796740 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796747 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796754 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796759 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796764 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796768 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796773 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796777 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796781 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796786 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796790 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796794 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796800 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796804 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796808 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796813 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.809749 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796817 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796821 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796827 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796831 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796835 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796840 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796844 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796848 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796852 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796856 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796860 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796866 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796871 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796875 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796880 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796884 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796888 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796892 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796897 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796901 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.810317 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796905 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796909 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796913 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796917 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796922 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796926 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796930 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796935 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796939 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796944 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796949 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796953 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796957 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796961 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796967 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796972 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796976 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796981 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.796985 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797006 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.810853 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797011 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797015 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797019 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797024 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797028 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797033 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797037 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797041 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797046 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797050 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797057 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797062 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797067 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797072 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797077 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797081 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797086 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797090 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797095 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.811511 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797099 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797103 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797107 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797112 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797117 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797121 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797126 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797133 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797137 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.797141 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.802473 6481 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.802489 6481 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.802507 6481 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.802512 6481 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802574 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802580 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802583 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802587 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802590 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802593 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.811961 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802596 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802599 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802602 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802605 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802608 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802610 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802613 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802616 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802618 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802621 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802623 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802626 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802629 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802631 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802633 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802636 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802639 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802642 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802645 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802648 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.812453 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802651 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802654 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802656 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802659 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802662 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802664 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802667 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802669 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802672 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802675 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802678 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802688 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802691 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802694 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802696 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802699 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802702 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802705 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802708 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.812930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802712 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802722 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802725 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802728 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802731 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802734 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802737 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802739 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802742 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802745 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802747 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802750 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802753 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802756 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802758 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802761 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802764 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802768 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802772 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.813451 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802775 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802778 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802781 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802783 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802786 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802788 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802791 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802794 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802796 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802800 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802803 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802805 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802808 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802811 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802813 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802821 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802824 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802828 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802833 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802837 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.813913 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802840 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802845 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802849 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802853 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802858 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802861 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802864 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802866 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802870 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802872 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802874 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802877 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802880 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802883 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802886 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802888 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802891 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802893 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802896 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.814413 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802898 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802901 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.802903 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.802932 6481 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803084 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803089 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803092 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803095 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803097 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803106 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803109 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803112 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803115 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803117 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803121 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803125 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.814881 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803130 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803134 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803138 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803141 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803146 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803150 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803154 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803157 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803160 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803162 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803165 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803167 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803170 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803172 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803175 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803177 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803180 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803183 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803186 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.815390 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803188 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803191 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803193 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803196 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803198 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803201 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803204 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803215 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803218 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803220 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803223 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803226 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803228 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803231 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803234 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803236 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803239 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803241 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803244 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.816139 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803247 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803251 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803254 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803257 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803259 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803262 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803264 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803267 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803269 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803272 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803275 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803277 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803280 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803282 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803285 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803287 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803290 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803293 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803295 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.816705 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803299 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803302 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803311 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803315 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803318 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803320 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803323 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803325 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803328 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803331 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803334 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803336 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803339 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803341 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803344 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803347 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803349 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803351 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803354 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803357 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.817177 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803359 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803361 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803364 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803366 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803369 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803371 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803374 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803376 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803379 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803381 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803384 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803387 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803389 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803392 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803394 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803400 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.817670 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803403 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.803429 6481 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803462 6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803466 6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803471 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803474 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803478 6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803480 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803483 6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803486 6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803489 6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803491 6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803494 6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803496 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803499 6481 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803503 6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:17:08.818061 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803507 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803510 6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803513 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803516 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803518 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803521 6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803523 6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803526 6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803529 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803532 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803534 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803537 6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803540 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803543 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803545 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803548 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803553 6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803555 6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803558 6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803561 6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:17:08.818448 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803565 6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803567 6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803570 6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803573 6481 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803575 6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803578 6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803580 6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803583 6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803586 6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803589 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803591 6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803594 6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803597 6481 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803599 6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803602 6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803605 6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803607 6481 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803610 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803612 6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803615 6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:17:08.818930 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803618 6481 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803620 6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803623 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803626 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803628 6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803631 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803633 6481 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803636 6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803640 6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803644 6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803646 6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803649 6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803653 6481 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803655 6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803659 6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803664 6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803668 6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803672 6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803676 6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803679 6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:17:08.819461 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803683 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803687 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803692 6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803696 6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803700 6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803704 6481 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803707 6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803710 6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803712 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803715 6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803717 6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803720 6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803723 6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803726 6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803729 6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803731 6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803734 6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803737 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803739 6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:17:08.819927 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803742 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803746 6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803749 6481 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803752 6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803755 6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803759 6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803761 6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803766 6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803770 6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803772 6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803775 6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803778 6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:08.803781 6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.803856 6481 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.804899 6481 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.804954 6481 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.805137 6481 server.go:1014] "Starting client certificate rotation" Oct 31 19:17:08.820563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.805273 6481 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.805322 6481 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:10:56 +0000 UTC" deadline="2028-07-03 15:52:57.278145056 +0000 UTC" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.805358 6481 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14660h35m48.472788937s" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.805584 6481 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.806645 6481 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.807739 6481 log.go:25] "Validated CRI v1 runtime API" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.809603 6481 log.go:25] "Validated CRI v1 image API" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.810329 6481 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 19:17:08.820975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.812348 6481 fs.go:135] Filesystem UUIDs: map[637e06d7-878b-4ba0-90a0-cd5e63e8eba5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c4a26a6c-9f1b-49b1-af84-246f59656ed8:/dev/nvme0n1p4] Oct 31 19:17:08.821235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.812371 6481 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0687ff29840bcc8977d03289bcd53f416253d7ffa5c13bc787818c3cccce34e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0687ff29840bcc8977d03289bcd53f416253d7ffa5c13bc787818c3cccce34e5/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/253fd3cc9fdf1db01448e788d889783ed16d402e5807040136ddfe322a326763/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/253fd3cc9fdf1db01448e788d889783ed16d402e5807040136ddfe322a326763/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31909f6b80b3f5b5d87fd4ee5e839eec6d2d02837e0f124f826a955eeff82d27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31909f6b80b3f5b5d87fd4ee5e839eec6d2d02837e0f124f826a955eeff82d27/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/451293960e262de2f878c58e3f22f1ab4b7b2bf73b73841ecb19970de73be4f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/451293960e262de2f878c58e3f22f1ab4b7b2bf73b73841ecb19970de73be4f4/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49f976ab4da4a06fc84b5b25b75e6423f07c857f74012908d5cade0b3602498b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49f976ab4da4a06fc84b5b25b75e6423f07c857f74012908d5cade0b3602498b/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4adbe3b5384c7b099b65ddd42f82dcc02935e7827d9710f01d79c48a17851017/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4adbe3b5384c7b099b65ddd42f82dcc02935e7827d9710f01d79c48a17851017/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5865a98fd665c45f048f95b821958a0d8c60f51a41c92c3a051407a709acfd24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5865a98fd665c45f048f95b821958a0d8c60f51a41c92c3a051407a709acfd24/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6620411e297a1abdfe5e246dcc8b49659358268e55b1a02ad2df65314458b27d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6620411e297a1abdfe5e246dcc8b49659358268e55b1a02ad2df65314458b27d/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ee3f2b1133b0df17678dd02b3a53b0ac45019a9bba2b8a5b70d45ab0a72b39a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ee3f2b1133b0df17678dd02b3a53b0ac45019a9bba2b8a5b70d45ab0a72b39a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9351ffa9e461c22db0c9cf0e5f2904897c0364879d7ab95694ff29d189ccbd78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9351ffa9e461c22db0c9cf0e5f2904897c0364879d7ab95694ff29d189ccbd78/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9658c19b90f33dc67e39586a7deb0982684d3416c176c8226b9126fa2e5e3c9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9658c19b90f33dc67e39586a7deb0982684d3416c176c8226b9126fa2e5e3c9a/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/991eed3e6acf55afdbe8fc4215f40ee696edaae7e27b8fc657e3d400e3ed8ffc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/991eed3e6acf55afdbe8fc4215f40ee696edaae7e27b8fc657e3d400e3ed8ffc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99b4ec3ec3b56b658df709725abf3a8191f2b0a15dd27d6eee2d3b5eec598024/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99b4ec3ec3b56b658df709725abf3a8191f2b0a15dd27d6eee2d3b5eec598024/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aafd637df84a04707715c59df86cb8923ab7de1ca0d4ee0e3c02170a5b431bfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aafd637df84a04707715c59df86cb8923ab7de1ca0d4ee0e3c02170a5b431bfc/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0195ffa98da70adaf08c1f742bbbff32bb6e97f6837a23a1d9c22449a0fb379/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0195ffa98da70adaf08c1f742bbbff32bb6e97f6837a23a1d9c22449a0fb379/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de47d30b150997640b6df36ad10c23589238c8a4ff1e529238fce29e26d78f5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de47d30b150997640b6df36ad10c23589238c8a4ff1e529238fce29e26d78f5b/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecb3cc6249324da9b95c5d1db8b4388fe0047431a99d201ed00b719d79e4ced8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecb3cc6249324da9b95c5d1db8b4388fe0047431a99d201ed00b719d79e4ced8/userdata/shm major:0 minor:382 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0a7ad93fb0798da154a237136724b6a24d4d754ed38c3e7b6c5233f52fc30a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0a7ad93fb0798da154a237136724b6a24d4d754ed38c3e7b6c5233f52fc30a3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f46f1c6d3380aa4ff141fa2d144bb00ebf76419c1459c6c272debffc3fa49b34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f46f1c6d3380aa4ff141fa2d144bb00ebf76419c1459c6c272debffc3fa49b34/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6/volumes/kubernetes.io~projected/kube-api-access-x9flg:{mountpoint:/var/lib/kubelet/pods/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6/volumes/kubernetes.io~projected/kube-api-access-x9flg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc/volumes/kubernetes.io~projected/kube-api-access-44xg2:{mountpoint:/var/lib/kubelet/pods/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc/volumes/kubernetes.io~projected/kube-api-access-44xg2 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f90009c-b0e7-4288-ac5f-2a5fdc462b18/volumes/kubernetes.io~projected/kube-api-access-zz4rl:{mountpoint:/var/lib/kubelet/pods/0f90009c-b0e7-4288-ac5f-2a5fdc462b18/volumes/kubernetes.io~projected/kube-api-access-zz4rl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14c0ff06-8471-4aff-b1a2-ab2450b62cec/volumes/kubernetes.io~projected/kube-api-access-m58vt:{mountpoint:/var/lib/kubelet/pods/14c0ff06-8471-4aff-b1a2-ab2450b62cec/volumes/kubernetes.io~projected/kube-api-access-m58vt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~projected/kube-api-access-tt7g2:{mountpoint:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~projected/kube-api-access-tt7g2 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/280701d1-1545-42b0-bb86-39832f46f516/volumes/kubernetes.io~projected/kube-api-access-4zssg:{mountpoint:/var/lib/kubelet/pods/280701d1-1545-42b0-bb86-39832f46f516/volumes/kubernetes.io~projected/kube-api-access-4zssg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/280701d1-1545-42b0-bb86-39832f46f516/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/280701d1-1545-42b0-bb86-39832f46f516/volumes/kubernetes.io~secret/metrics-certs major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c915558-1864-4e8a-8fdf-89439548044c/volumes/kubernetes.io~projected/kube-api-access-86rpc:{mountpoint:/var/lib/kubelet/pods/3c915558-1864-4e8a-8fdf-89439548044c/volumes/kubernetes.io~projected/kube-api-access-86rpc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volumes/kubernetes.io~projected/kube-api-access-sd2kr:{mountpoint:/var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volumes/kubernetes.io~projected/kube-api-access-sd2kr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61f4f0c2-7d69-487c-bfd0-cc4e5da89716/volumes/kubernetes.io~projected/kube-api-access-xs5rd:{mountpoint:/var/lib/kubelet/pods/61f4f0c2-7d69-487c-bfd0-cc4e5da89716/volumes/kubernetes.io~projected/kube-api-access-xs5rd major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61f4f0c2-7d69-487c-bfd0-cc4e5da89716/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/61f4f0c2-7d69-487c-bfd0-cc4e5da89716/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~projected/kube-api-access-vjtmz:{mountpoint:/var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~projected/kube-api-access-vjtmz major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84852974-1373-452c-b758-6a5e03bc5ac6/volumes/kubernetes.io~projected/kube-api-access-mxr72:{mountpoint:/var/lib/kubelet/pods/84852974-1373-452c-b758-6a5e03bc5ac6/volumes/kubernetes.io~projected/kube-api-access-mxr72 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84852974-1373-452c-b758-6a5e03bc5ac6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/84852974-1373-452c-b758-6a5e03bc5ac6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~projected/kube-api-access-f8pcl:{mountpoint:/var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~projected/kube-api-access-f8pcl major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~projected/kube-api-access-6vhzf:{mountpoint:/var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~projected/kube-api-access-6vhzf major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~projected/kube-api-access-f66xx:{mountpoint:/var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~projected/kube-api-access-f66xx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9976ed46-0a87-46f9-982d-4a4ac667e834/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9976ed46-0a87-46f9-982d-4a4ac667e834/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~projected/kube-api-access-74g88:{mountpoint:/var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~projected/kube-api-access-74g88 major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~projected/kube-api-access-7r8n7:{mountpoint:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~projected/kube-api-access-7r8n7 major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e867aa15-55b5-48e1-8424-f5b176c3bcdb/volumes/kubernetes.io~projected/kube-api-access-qzq4s:{mountpoint:/var/lib/kubelet/pods/e867aa15-55b5-48e1-8424-f5b176c3bcdb/volumes/kubernetes.io~projected/kube-api-access-qzq4s major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9968b83-2472-46f7-9ca4-664a3b8f89cd/volumes/kubernetes.io~projected/kube-api-access-gktr6:{mountpoint:/var/lib/kubelet/pods/e9968b83-2472-46f7-9ca4-664a3b8f89cd/volumes/kubernetes.io~projected/kube-api-access-gktr6 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9968b83-2472-46f7-9ca4-664a3b8f89cd/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e9968b83-2472-46f7-9ca4-664a3b8f89cd/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efc46de0-c9f5-4659-9713-631a24271f0d/volumes/kubernetes.io~projected/kube-api-access-6n2zr:{mountpoint:/var/lib/kubelet/pods/efc46de0-c9f5-4659-9713-631a24271f0d/volumes/kubernetes.io~projected/kube-api-access-6n2zr major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5aaedfd-3514-492d-96b2-2dc095ff2228/volumes/kubernetes.io~projected/kube-api-access-gl8vc:{mountpoint:/var/lib/kubelet/pods/f5aaedfd-3514-492d-96b2-2dc095ff2228/volumes/kubernetes.io~projected/kube-api-access-gl8vc major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb981019-7d15-475e-ac02-ef33f915867a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fb981019-7d15-475e-ac02-ef33f915867a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:375 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/643d2fcb46ea83fb4b81931fbc9baead4d886c1327e0d4925545e5831ec619cf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3f486d3ba343e14bb66d34bb88100cb2c69f456ad192e92611b3b1cf066415b9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/18b85c814e8cfc252678b1d815b52e75c07246976589b69ff13802f4d8340753/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d855cd986ee75633c9f08636f93562fa4d72db8c74235f115ec6ecd9a29a0077/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b9cb4bd032e943a5260964347efe462389ae35633363f1a9cdb74fd77861c437/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4a3eec0e3dc7d82c1ec64e3aafc2d869b08fd74613dc81959bf4e93a4dccfbc4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a5fa214017d7f19cfcff8a65b26ecd040d7c444c84d1b10fdc01a914a4f3b6c8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2a9ffc8325290927d6e9a2cdde7430f29302b4143528ff791c86b8a0507c13f8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8df177cfbc4c687e9365f16691a1e613e25dc843bb1d35f78e8b0750329b8648/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/712f0553c125ab2ed462e5bcc9cf52020e0074de06771059f733f44cf02a9848/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/55f501e5928fdc3837b156da9b34799e82506c1a8f47456c356e120e1e535c60/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/79f936bb3501e645dabcf91d41a5f217c7f759539e2d923ffbfa5989b5052ac0/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/65e55e03dddc23176eec5dbaf7e5d7ccfe5c8e85b7952dc2e7ec8be1d5d0feb4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e7ed53799d46c9aa301a3de837165d2907b37d7e81ce298bcc703daf099ea40c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f1796e811fc6418550c2bc851b16ae341f6967da2562505566c7ef76c128382f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/330bc0761fcb3145a64e49698bf671563e2b97fb3f1fef43af3533083da72d33/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2a6b2bf11a7d6262f409a00768f632940ad17acdad66686daff0868aa21c45fa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/31fac7d177533bde88751b4f352eaeb21901f05cfd891fa210ca17111378a3d6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/79c5b6e9890ba8ddbf867d279c23c868d57517d8c5f47c79fa6fd60a38d6143c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/eef1f7f2338fc34e97392e25ee11c27f05c0cfa488d4ecdb79879d306256e09f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/356b9da3ded25d36a202c663b8703581955262a163b216f4d67465dbaeb81291/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/85f6c3c605e78f4e52866aa2796a690609721c4209398d5b0ab60269b6652b50/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8e4df05cf9711c0e8e0e07c30791d0c5200cb56c42ed4b37be91394a95e4b2e6/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e0089fd5039cacc8167d4635e41382307a7bca5709847075dbd6ddedda0b5e4d/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/cea4e9c78308ce2b82b168ea629e2d92350cac42e75ec3ac30d40fb61fc56780/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/9a4eda2ade4f8ab9f83413b7eb54a070e64ed58e3e1a41696a602222839a8ce2/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/ae5698f2610b7a8665a2c4d3e7ce790ccdb17e028dde4f8c544e607f84d5396f/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/b4038cc5403a06ad6b064bea6641f7d164bb27ebf4106a6c514df8f2f7ec3a75/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/61a43a0fa0204b7449cef905e87fef1c7903b8fd99a61b29387a9dd52aadea39/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/0ad6153f384c4ec1c95205293c8c54e1b622bcf51d8cd22cda5a3d7366a34ee5/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/58248325e3789003fee3a9a2b82c8dfa5d91ace936c7aa44d30c9478d45f096c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/0b94b2a2a2158e338b589ee4b6257004fd5d8d9c210d66cc16b734ee689081b3/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/4eae99b4de847b59436a0ebf02ffdfbbaa96a513b6bd844b21615a19275630a4/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/df0822b8494acccbd555599f1702cc774a479a56157bc38ebba90baea6fde6fc/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/d4949e76779709d30bc8509844a861b12fc93c2e11c76ddea23f70361d757a21/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/49466901646bff163fb5780fb71f811c056b69ff7ec479547eb2c81e0a9ad2f3/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/51d113f2aaabe194d36209ede6d501e5700d59aaf8f3c3e88b7002d0578d36ea/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/15009ea6b50d6b46777ee9825af6412ca2eadd64057f169723233818af800a41/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/762e69a97b57ef415a92d179d7561b9a479b256ba8a5a68e4c6a39e31ca0b6dd/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/1c9c22dc36838eb9807d0d424a2dbc5b26c86947f354ced4ae5de87408d93a66/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/5aa5c6407c94a1e0eafd8ed3a8209d67ba75ec2378177b01a432919427fc04e1/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/960d5802deabf55d808fdbe3f1289e5af6aa2a32731f43f76e207b3b171552dc/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/18ec4de07927855fb07f59cfa068b8f46abd0acc9ccbd0c50a9f5d2f53bde194/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/e3aab173ec4bf4cc6629e6582796a360aa779c8422e2609cee9d31c622ddd215/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/adb3f0d0c2d3e13d22ad56d6d3a6baa4b977b218c8f6a632e827a0301f7226e2/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/7af6582b2d8702d61759235de1fa5af2ca4b70e8bcecad56312272110b971f71/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/94a854fc020f56f7e4f5ee747ed88d4e38a77250dbfa91483fdf515a4aa4d2f4/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/c3ea381a2640d05cf308828b1394653353637e43a51e3154b77936353160e2a5/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/f1fd9f5ccf32d3e16dcd39cc7316235bde6be896034ad05cdf97700b50f4386e/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/80d4f12834c43f078fb784f3794a99aaf00e2ab75ff4c92bed7ac6a7a5b98ba2/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/552dc52fec237e3fd8911943df0f322c862c038b5b46d0e608297851db117631/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/7a804af6e14322964460db4f0ab83eaa04689f7093b8f7320c8c1cecdb28a38c/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/cd06e4a67764fc43b7335acd1206e4218700d2932755be87b519ea1ac83fd67a/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/7f25d86eaefa33c2227a0758a371ae9a95ff0f761fe1dd2112673afad208a137/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/4caf60a322377e6d8b220bb857be86f942e8b81f1e58a05dcc1a06fc7699a010/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/0806e325c4392ef90ad8ab3757e3dbdf718964247b5e83fde9ca1aaf2c66d491/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/e14cf1479ca73ae524c0860edc0e263028b954d07d8ade4dd33db01d2de61ac3/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/a1ae1ae5d89c7adccde4837fc0dad84411fe9aba5181b16ed3f482238ed82b57/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/4809d0858d28e62df745e640f02886a0bb32b0a8f5184bda532ff2823386c402/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f6383b2a72b4ca3b91a13f39a1e10d6302f66e3b0b467448144591b0f671cdaa/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/b9d83dc77a286c3c77e2738f4f6287d51ce9482fc3f42fb4c25e563b2d19f2ca/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/1dce81e3df0e3f10dda51485b003b8486d1816284a89944ef4da2f02a36b561a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/0c2b91b27ce46699ac0402526ef2860e0c5d42ec22a4df519389f3e55af33c13/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/bddb67eeaa2f073e59d3e0f55056585c7a9cb9a4c027cee128a321492414ffe7/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/363fd1302c5ba23f1a16d77549b319e263b8439e69c12a6aaebabfe3700c11a3/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cde3b6e3c30e0e7dfd9d8f871ba17140016937e7427ca1cb7847f1f7836fff91/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fcff11366996414cf653ad007b058361a7c06c0c2e1d65458d82782a71496989/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/40372b17642a9c5127c3c710e1d587e27cf6ce4ed256c2c1e6dc06f5ab967ba3/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.820568 6481 manager.go:217] Machine: {Timestamp:2026-10-31 19:17:08.820150969 +0000 UTC m=+0.060374089 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3085413 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8178473ea861f53639d68e37a0a3 SystemUUID:ec2b8178-473e-a861-f536-39d68e37a0a3 BootID:42918202-70f1-40da-8662-6cde8669caf8 Filesystems:[{Device:/run/containers/storage/overlay-containers/31909f6b80b3f5b5d87fd4ee5e839eec6d2d02837e0f124f826a955eeff82d27/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9351ffa9e461c22db0c9cf0e5f2904897c0364879d7ab95694ff29d189ccbd78/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efc46de0-c9f5-4659-9713-631a24271f0d/volumes/kubernetes.io~projected/kube-api-access-6n2zr DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49f976ab4da4a06fc84b5b25b75e6423f07c857f74012908d5cade0b3602498b/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:305 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0a7ad93fb0798da154a237136724b6a24d4d754ed38c3e7b6c5233f52fc30a3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecb3cc6249324da9b95c5d1db8b4388fe0047431a99d201ed00b719d79e4ced8/userdata/shm DeviceMajor:0 DeviceMinor:382 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~projected/kube-api-access-f8pcl DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:306 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84852974-1373-452c-b758-6a5e03bc5ac6/volumes/kubernetes.io~projected/kube-api-access-mxr72 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volumes/kubernetes.io~projected/kube-api-access-sd2kr DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9968b83-2472-46f7-9ca4-664a3b8f89cd/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f46f1c6d3380aa4ff141fa2d144bb00ebf76419c1459c6c272debffc3fa49b34/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:259 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c915558-1864-4e8a-8fdf-89439548044c/volumes/kubernetes.io~projected/kube-api-access-86rpc DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e867aa15-55b5-48e1-8424-f5b176c3bcdb/volumes/kubernetes.io~projected/kube-api-access-qzq4s DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84852974-1373-452c-b758-6a5e03bc5ac6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~projected/kube-api-access-7r8n7 DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99b4ec3ec3b56b658df709725abf3a8191f2b0a15dd27d6eee2d3b5eec598024/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14c0ff06-8471-4aff-b1a2-ab2450b62cec/volumes/kubernetes.io~projected/kube-api-access-m58vt DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fc92bb3-b6ec-4407-b9dd-6082981560e6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/61f4f0c2-7d69-487c-bfd0-cc4e5da89716/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/451293960e262de2f878c58e3f22f1ab4b7b2bf73b73841ecb19970de73be4f4/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~projected/kube-api-access-tt7g2 DeviceMajor:0 DeviceMinor:298 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6620411e297a1abdfe5e246dcc8b49659358268e55b1a02ad2df65314458b27d/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0687ff29840bcc8977d03289bcd53f416253d7ffa5c13bc787818c3cccce34e5/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc/volumes/kubernetes.io~projected/kube-api-access-44xg2 DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de47d30b150997640b6df36ad10c23589238c8a4ff1e529238fce29e26d78f5b/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/280701d1-1545-42b0-bb86-39832f46f516/volumes/kubernetes.io~projected/kube-api-access-4zssg DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~projected/kube-api-access-6vhzf DeviceMajor:0 DeviceMinor:362 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9976ed46-0a87-46f9-982d-4a4ac667e834/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9658c19b90f33dc67e39586a7deb0982684d3416c176c8226b9126fa2e5e3c9a/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61f4f0c2-7d69-487c-bfd0-cc4e5da89716/volumes/kubernetes.io~projected/kube-api-access-xs5rd DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ee3f2b1133b0df17678dd02b3a53b0ac45019a9bba2b8a5b70d45ab0a72b39a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/280701d1-1545-42b0-bb86-39832f46f516/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:380 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0f90009c-b0e7-4288-ac5f-2a5fdc462b18/volumes/kubernetes.io~projected/kube-api-access-zz4rl DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~projected/kube-api-access-f66xx DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5865a98fd665c45f048f95b821958a0d8c60f51a41c92c3a051407a709acfd24/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/991eed3e6acf55afdbe8fc4215f40ee696edaae7e27b8fc657e3d400e3ed8ffc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aafd637df84a04707715c59df86cb8923ab7de1ca0d4ee0e3c02170a5b431bfc/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fb981019-7d15-475e-ac02-ef33f915867a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df7cc259-0fb7-4f12-ac63-1b798ada6303/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:307 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/945c708d-ed9a-496a-8cfb-2531e8b9a15a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e993f09-bc04-46f4-9c3a-c77ba0341b81/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0195ffa98da70adaf08c1f742bbbff32bb6e97f6837a23a1d9c22449a0fb379/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:361 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~projected/kube-api-access-74g88 DeviceMajor:0 DeviceMinor:264 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/264ad1b2-15f9-4b71-97ef-96b7f380c650/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73474282-1f87-4204-969d-2ddcbb5ee758/volumes/kubernetes.io~projected/kube-api-access-vjtmz DeviceMajor:0 DeviceMinor:271 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/253fd3cc9fdf1db01448e788d889783ed16d402e5807040136ddfe322a326763/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f5aaedfd-3514-492d-96b2-2dc095ff2228/volumes/kubernetes.io~projected/kube-api-access-gl8vc DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6/volumes/kubernetes.io~projected/kube-api-access-x9flg DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9968b83-2472-46f7-9ca4-664a3b8f89cd/volumes/kubernetes.io~projected/kube-api-access-gktr6 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:263 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4adbe3b5384c7b099b65ddd42f82dcc02935e7827d9710f01d79c48a17851017/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0687ff29840bcc8 MacAddress:96:a3:34:bf:35:aa Speed:10000 Mtu:8901} {Name:283b9d33df5fef7 MacAddress:c6:a1:09:ae:88:0a Speed:10000 Mtu:8901} {Name:31cc8052b06bb42 MacAddress:e6:0e:36:6a:de:df Speed:10000 Mtu:8901} {Name:5865a98fd665c45 MacAddress:9e:6b:6c:43:de:6a Speed:10000 Mtu:8901} {Name:6620411e297a1ab MacAddress:ae:cc:60:c8:e7:cc Speed:10000 Mtu:8901} {Name:8ee3f2b1133b0df MacAddress:fe:cc:c1:41:0a:78 Speed:10000 Mtu:8901} {Name:9658c19b90f33dc MacAddress:12:f4:07:e5:7d:e2 Speed:10000 Mtu:8901} {Name:99b4ec3ec3b56b6 MacAddress:be:8c:a9:62:a9:df Speed:10000 Mtu:8901} {Name:b0195ffa98da70a MacAddress:a2:c1:95:f8:45:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fb:65:9a:6d Speed:0 Mtu:9001} {Name:br-int MacAddress:26:32:da:d5:e8:e6 Speed:0 Mtu:8901} {Name:de47d30b1509976 MacAddress:fa:7d:15:cf:ad:01 Speed:10000 Mtu:8901} {Name:ecb3cc6249324da MacAddress:d2:1f:03:ef:10:57 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fb:65:9a:6d Speed:0 Mtu:9001} {Name:f46f1c6d3380aa4 MacAddress:0a:64:01:fa:65:47 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:aa:ed:d0:92:5a:16 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:7f:05:82:dd:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.820937 6481 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.820981 6481 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821287 6481 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821308 6481 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-134.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821426 6481 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821433 6481 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821444 6481 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:17:08.821497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821457 6481 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821522 6481 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821697 6481 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821798 6481 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821833 6481 kubelet.go:493] "Attempting to sync node with API server" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821848 6481 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821857 6481 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821866 6481 kubelet.go:405] "Adding apiserver pod source" Oct 31 19:17:08.821885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.821874 6481 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 19:17:08.822812 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.822797 6481 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 19:17:08.822849 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.822814 6481 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:17:08.823296 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.823281 6481 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 19:17:08.825665 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.825653 6481 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 19:17:08.825727 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.825666 6481 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:17:08.825757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.825751 6481 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 19:17:08.825924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.825914 6481 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 19:17:08.825954 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.825942 6481 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 19:17:08.826057 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826047 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 19:17:08.826093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826061 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 19:17:08.826093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826068 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 19:17:08.826093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826073 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 19:17:08.826093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826079 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 19:17:08.826093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826084 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 19:17:08.826093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826090 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 19:17:08.826093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826095 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 19:17:08.826282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826103 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 19:17:08.826282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826108 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 19:17:08.826282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826116 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 19:17:08.826282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826126 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 19:17:08.826282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826150 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 19:17:08.826282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.826160 6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 19:17:08.829391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.829373 6481 server.go:1263] "Started kubelet" Oct 31 19:17:08.829486 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.829432 6481 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 19:17:08.829564 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.829512 6481 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 19:17:08.829611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.829586 6481 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 19:17:08.829860 ip-10-0-143-134 systemd[1]: Started Kubernetes Kubelet. Oct 31 19:17:08.830020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.829984 6481 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 19:17:08.831157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.831140 6481 server.go:317] "Adding debug handlers to kubelet server" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.835964 6481 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.835985 6481 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.836051 6481 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:10:58 +0000 UTC" deadline="2028-07-01 14:44:12.985511324 +0000 UTC" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.836067 6481 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14611h27m4.149446863s" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.836083 6481 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.836095 6481 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:08.836106 6481 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:17:08.836529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.836195 6481 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 19:17:08.837458 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837436 6481 factory.go:153] Registering CRI-O factory Oct 31 19:17:08.837559 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837462 6481 factory.go:223] Registration of the crio container factory successfully Oct 31 19:17:08.837559 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837516 6481 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 19:17:08.837559 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837526 6481 factory.go:55] Registering systemd factory Oct 31 19:17:08.837559 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837535 6481 factory.go:223] Registration of the systemd container factory successfully Oct 31 19:17:08.837559 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837552 6481 factory.go:103] Registering Raw factory Oct 31 19:17:08.837781 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837566 6481 manager.go:1199] Started watching for new ooms in manager Oct 31 19:17:08.837928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.837914 6481 manager.go:319] Starting recovery of all containers Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838776 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc" volumeName="kubernetes.io/projected/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-kube-api-access-44xg2" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838814 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838828 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838842 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c915558-1864-4e8a-8fdf-89439548044c" volumeName="kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-cni-binary-copy" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838853 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7cc259-0fb7-4f12-ac63-1b798ada6303" volumeName="kubernetes.io/projected/df7cc259-0fb7-4f12-ac63-1b798ada6303-kube-api-access-7r8n7" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838865 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c0ff06-8471-4aff-b1a2-ab2450b62cec" volumeName="kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838878 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="280701d1-1545-42b0-bb86-39832f46f516" volumeName="kubernetes.io/projected/280701d1-1545-42b0-bb86-39832f46f516-kube-api-access-4zssg" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838891 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="945c708d-ed9a-496a-8cfb-2531e8b9a15a" volumeName="kubernetes.io/projected/945c708d-ed9a-496a-8cfb-2531e8b9a15a-kube-api-access-f66xx" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838905 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/empty-dir/264ad1b2-15f9-4b71-97ef-96b7f380c650-tmp" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838919 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838931 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838942 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4d70a7b-5d24-4d0f-909e-56b6b8273060" volumeName="kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838954 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e867aa15-55b5-48e1-8424-f5b176c3bcdb" volumeName="kubernetes.io/configmap/e867aa15-55b5-48e1-8424-f5b176c3bcdb-iptables-alerter-script" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838967 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9968b83-2472-46f7-9ca4-664a3b8f89cd" volumeName="kubernetes.io/projected/e9968b83-2472-46f7-9ca4-664a3b8f89cd-kube-api-access-gktr6" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.838982 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6" volumeName="kubernetes.io/empty-dir/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-tmp-dir" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839008 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839019 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9976ed46-0a87-46f9-982d-4a4ac667e834" volumeName="kubernetes.io/configmap/9976ed46-0a87-46f9-982d-4a4ac667e834-konnectivity-ca" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839032 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c915558-1864-4e8a-8fdf-89439548044c" volumeName="kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-multus-daemon-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839045 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73474282-1f87-4204-969d-2ddcbb5ee758" volumeName="kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839058 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84852974-1373-452c-b758-6a5e03bc5ac6" volumeName="kubernetes.io/configmap/84852974-1373-452c-b758-6a5e03bc5ac6-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839069 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839082 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/configmap/264ad1b2-15f9-4b71-97ef-96b7f380c650-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839093 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73474282-1f87-4204-969d-2ddcbb5ee758" volumeName="kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839105 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839119 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="945c708d-ed9a-496a-8cfb-2531e8b9a15a" volumeName="kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-tuned" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839130 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4d70a7b-5d24-4d0f-909e-56b6b8273060" volumeName="kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839142 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb981019-7d15-475e-ac02-ef33f915867a" volumeName="kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839154 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc" volumeName="kubernetes.io/secret/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839171 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61f4f0c2-7d69-487c-bfd0-cc4e5da89716" volumeName="kubernetes.io/projected/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-kube-api-access-xs5rd" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839183 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73474282-1f87-4204-969d-2ddcbb5ee758" volumeName="kubernetes.io/empty-dir/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-textfile" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839195 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839207 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7cc259-0fb7-4f12-ac63-1b798ada6303" volumeName="kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839218 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7cc259-0fb7-4f12-ac63-1b798ada6303" volumeName="kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-client-ca-bundle" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839232 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c0ff06-8471-4aff-b1a2-ab2450b62cec" volumeName="kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-binary-copy" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839245 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c0ff06-8471-4aff-b1a2-ab2450b62cec" volumeName="kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839258 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9976ed46-0a87-46f9-982d-4a4ac667e834" volumeName="kubernetes.io/secret/9976ed46-0a87-46f9-982d-4a4ac667e834-agent-certs" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839268 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7cc259-0fb7-4f12-ac63-1b798ada6303" volumeName="kubernetes.io/empty-dir/df7cc259-0fb7-4f12-ac63-1b798ada6303-audit-log" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839280 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9968b83-2472-46f7-9ca4-664a3b8f89cd" volumeName="kubernetes.io/empty-dir/e9968b83-2472-46f7-9ca4-664a3b8f89cd-tmp-dir" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839291 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9968b83-2472-46f7-9ca4-664a3b8f89cd" volumeName="kubernetes.io/secret/e9968b83-2472-46f7-9ca4-664a3b8f89cd-metrics-tls" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839302 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839314 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc92bb3-b6ec-4407-b9dd-6082981560e6" volumeName="kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-env-overrides" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839340 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/projected/264ad1b2-15f9-4b71-97ef-96b7f380c650-kube-api-access-tt7g2" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839352 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839365 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="280701d1-1545-42b0-bb86-39832f46f516" volumeName="kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839377 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4d70a7b-5d24-4d0f-909e-56b6b8273060" volumeName="kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839388 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e867aa15-55b5-48e1-8424-f5b176c3bcdb" volumeName="kubernetes.io/projected/e867aa15-55b5-48e1-8424-f5b176c3bcdb-kube-api-access-qzq4s" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839403 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-grpc-tls" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839414 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73474282-1f87-4204-969d-2ddcbb5ee758" volumeName="kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839428 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839442 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4d70a7b-5d24-4d0f-909e-56b6b8273060" volumeName="kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839454 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="264ad1b2-15f9-4b71-97ef-96b7f380c650" volumeName="kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839465 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc92bb3-b6ec-4407-b9dd-6082981560e6" volumeName="kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839478 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73474282-1f87-4204-969d-2ddcbb5ee758" volumeName="kubernetes.io/projected/73474282-1f87-4204-969d-2ddcbb5ee758-kube-api-access-vjtmz" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839489 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9968b83-2472-46f7-9ca4-664a3b8f89cd" volumeName="kubernetes.io/configmap/e9968b83-2472-46f7-9ca4-664a3b8f89cd-config-volume" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839501 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6" volumeName="kubernetes.io/projected/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-kube-api-access-x9flg" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839515 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14c0ff06-8471-4aff-b1a2-ab2450b62cec" volumeName="kubernetes.io/projected/14c0ff06-8471-4aff-b1a2-ab2450b62cec-kube-api-access-m58vt" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839534 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c915558-1864-4e8a-8fdf-89439548044c" volumeName="kubernetes.io/projected/3c915558-1864-4e8a-8fdf-89439548044c-kube-api-access-86rpc" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839547 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61f4f0c2-7d69-487c-bfd0-cc4e5da89716" volumeName="kubernetes.io/secret/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-cert" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839559 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84852974-1373-452c-b758-6a5e03bc5ac6" volumeName="kubernetes.io/projected/84852974-1373-452c-b758-6a5e03bc5ac6-kube-api-access-mxr72" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.839571 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4d70a7b-5d24-4d0f-909e-56b6b8273060" volumeName="kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841714 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7cc259-0fb7-4f12-ac63-1b798ada6303" volumeName="kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841737 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5aaedfd-3514-492d-96b2-2dc095ff2228" volumeName="kubernetes.io/projected/f5aaedfd-3514-492d-96b2-2dc095ff2228-kube-api-access-gl8vc" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841761 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc92bb3-b6ec-4407-b9dd-6082981560e6" volumeName="kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-script-lib" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841774 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc92bb3-b6ec-4407-b9dd-6082981560e6" volumeName="kubernetes.io/projected/4fc92bb3-b6ec-4407-b9dd-6082981560e6-kube-api-access-sd2kr" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841799 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e993f09-bc04-46f4-9c3a-c77ba0341b81" volumeName="kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-tls" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841811 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4d70a7b-5d24-4d0f-909e-56b6b8273060" volumeName="kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841823 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7cc259-0fb7-4f12-ac63-1b798ada6303" volumeName="kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841837 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5aaedfd-3514-492d-96b2-2dc095ff2228" volumeName="kubernetes.io/configmap/f5aaedfd-3514-492d-96b2-2dc095ff2228-serviceca" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841849 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f90009c-b0e7-4288-ac5f-2a5fdc462b18" volumeName="kubernetes.io/projected/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kube-api-access-zz4rl" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841860 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc92bb3-b6ec-4407-b9dd-6082981560e6" volumeName="kubernetes.io/secret/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841873 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4d70a7b-5d24-4d0f-909e-56b6b8273060" volumeName="kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841885 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7cc259-0fb7-4f12-ac63-1b798ada6303" volumeName="kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841897 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="931e6634-fb0c-498e-b225-59a8deb17655" volumeName="kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841910 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73474282-1f87-4204-969d-2ddcbb5ee758" volumeName="kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-tls" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841922 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84852974-1373-452c-b758-6a5e03bc5ac6" volumeName="kubernetes.io/empty-dir/84852974-1373-452c-b758-6a5e03bc5ac6-data-volume" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841934 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84852974-1373-452c-b758-6a5e03bc5ac6" volumeName="kubernetes.io/secret/84852974-1373-452c-b758-6a5e03bc5ac6-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841946 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e993f09-bc04-46f4-9c3a-c77ba0341b81" volumeName="kubernetes.io/configmap/8e993f09-bc04-46f4-9c3a-c77ba0341b81-metrics-client-ca" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841958 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e993f09-bc04-46f4-9c3a-c77ba0341b81" volumeName="kubernetes.io/projected/8e993f09-bc04-46f4-9c3a-c77ba0341b81-kube-api-access-f8pcl" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841975 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e993f09-bc04-46f4-9c3a-c77ba0341b81" volumeName="kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.841989 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efc46de0-c9f5-4659-9713-631a24271f0d" volumeName="kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.842016 6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="945c708d-ed9a-496a-8cfb-2531e8b9a15a" volumeName="kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-tmp" seLinuxMountContext="" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.842029 6481 reconstruct.go:97] "Volume reconstruction finished" Oct 31 19:17:08.842688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.842037 6481 reconciler.go:29] "Reconciler: start to sync state" Oct 31 19:17:08.870181 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.870126 6481 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 19:17:08.871624 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.871600 6481 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 19:17:08.871624 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.871622 6481 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 19:17:08.871767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.871653 6481 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 19:17:08.871767 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:08.871696 6481 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 19:17:08.886192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.886161 6481 generic.go:356] "Generic (PLEG): container finished" podID="73474282-1f87-4204-969d-2ddcbb5ee758" containerID="f5c6111f4c8739484cdf49dcb86ea94ec8c27d2b44b95eb838f8b8b5c12c6262" exitCode=0 Oct 31 19:17:08.918291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.918256 6481 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="ce796ddf0b40d0922272a053f6615a9029d5672011dc9e687c026a4535bccc33" exitCode=0 Oct 31 19:17:08.918291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.918289 6481 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="75ea1b57cbefd4d7c7ab2adaaac607c0a067c8983c852f2b8f644905a54182d7" exitCode=0 Oct 31 19:17:08.918440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.918300 6481 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="e1c3a42bb23d25c948a2bc460361018cefa820035c8ab9f449b908e3995d87e8" exitCode=0 Oct 31 19:17:08.918440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.918311 6481 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="66643323355a2a716f56a6fc4ed398a57873bbc60bab6c7eb7666603ae705e2c" exitCode=0 Oct 31 19:17:08.918440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.918326 6481 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="93cd2a56b224b135da32a915e36b22ce5f4f328d77b6b95eb5e0c24301b2fa18" exitCode=0 Oct 31 19:17:08.918440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.918335 6481 generic.go:356] "Generic (PLEG): container finished" podID="14c0ff06-8471-4aff-b1a2-ab2450b62cec" containerID="eef1ac0cb7cda0c0fc2f09f1341cb7c1f41cd000178cc855902fe663785bb394" exitCode=0 Oct 31 19:17:08.931703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.931674 6481 generic.go:356] "Generic (PLEG): container finished" podID="d6c6a40b90966d73cbc8b2f5cf79a1c2" containerID="f8eac77e9a830b3a6f89d152d4012c28f7def79a9c0bfe8d6e458db05123db1a" exitCode=0 Oct 31 19:17:08.936190 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:08.936164 6481 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:17:08.939344 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.939327 6481 manager.go:324] Recovery completed Oct 31 19:17:08.957304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.957285 6481 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:17:08.959476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.959458 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:17:08.959565 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.959495 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:17:08.959565 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.959509 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:17:08.962947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.962922 6481 cpu_manager.go:226] "Starting" policy="none" Oct 31 19:17:08.962947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.962944 6481 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 19:17:08.963107 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.962966 6481 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 19:17:08.963210 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963171 6481 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 19:17:08.963210 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963194 6481 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 19:17:08.963363 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963220 6481 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 19:17:08.963363 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963230 6481 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 19:17:08.963497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963219 6481 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:17:08.963497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963463 6481 policy_none.go:50] "Start" Oct 31 19:17:08.963497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963489 6481 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 19:17:08.963622 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963505 6481 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 19:17:08.963770 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963755 6481 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 19:17:08.963829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963772 6481 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 19:17:08.963829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.963780 6481 policy_none.go:44] "Start" Oct 31 19:17:08.964816 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.964799 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:17:08.964898 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.964826 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:17:08.964898 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.964837 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:17:08.968842 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.968817 6481 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 19:17:08.972235 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:08.972184 6481 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 19:17:08.972315 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972283 6481 manager.go:341] "Starting Device Plugin manager" Oct 31 19:17:08.972315 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:08.972308 6481 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 19:17:08.972400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972320 6481 server.go:87] "Starting device plugin registration server" Oct 31 19:17:08.972583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972572 6481 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 19:17:08.972615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972583 6481 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 19:17:08.972684 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972668 6481 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 19:17:08.973562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972780 6481 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 19:17:08.973562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972822 6481 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 19:17:08.973562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972829 6481 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 19:17:08.973562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.972871 6481 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:17:08.97281164Z","UUID":"67952aa2-cfcd-4786-8c1d-9257d3233eb6","Handler":null,"Name":"","Endpoint":""} Oct 31 19:17:08.974638 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:08.974616 6481 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 19:17:08.974638 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:08.974656 6481 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-134.us-west-2.compute.internal\" not found" Oct 31 19:17:08.975390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.975362 6481 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 19:17:08.975480 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:08.975399 6481 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 19:17:09.015599 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.015551 6481 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:09.015727 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.015660 6481 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 19:17:09.016117 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.016105 6481 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:09.060938 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.060910 6481 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:09.073451 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.073427 6481 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:17:09.074564 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.074549 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:17:09.074692 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.074578 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:17:09.074692 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.074589 6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-134.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:17:09.074692 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.074630 6481 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.107343 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.107317 6481 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:17:09.172687 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.172636 6481 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal"] Oct 31 19:17:09.173246 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.173206 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"76ead912f9e2b59f12d28159dae3cb89","Type":"ContainerStarted","Data":"3b1b6c64097eb583676478354f62dfd8816be480c0a1f74e362354a36af1270f"} Oct 31 19:17:09.173353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.173260 6481 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.173353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.173250 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"76ead912f9e2b59f12d28159dae3cb89","Type":"ContainerStarted","Data":"1d92f655bef4483baebb5e8822e8132af90a30777cd03f18618075c3fcafe09e"} Oct 31 19:17:09.173353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.173347 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"d6c6a40b90966d73cbc8b2f5cf79a1c2","Type":"ContainerStarted","Data":"a9ec1b3f5409008362193202af4aed2675fce0d82b24b892a25cc6995eecb2b3"} Oct 31 19:17:09.173490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.173369 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"d6c6a40b90966d73cbc8b2f5cf79a1c2","Type":"ContainerDied","Data":"f8eac77e9a830b3a6f89d152d4012c28f7def79a9c0bfe8d6e458db05123db1a"} Oct 31 19:17:09.173490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.173389 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" event={"ID":"d6c6a40b90966d73cbc8b2f5cf79a1c2","Type":"ContainerStarted","Data":"53d6ae5000d0415eb1569db13d610eb58919c2c95c8bd90b36b85f739733c383"} Oct 31 19:17:09.173561 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.173504 6481 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.177739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.177725 6481 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.177793 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.177784 6481 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.230225 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.230161 6481 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:17:09.230225 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:09.230205 6481 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.230738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.230723 6481 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:17:09.230804 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:09.230754 6481 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.318628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.318602 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.318628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.318631 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.318830 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.318647 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76ead912f9e2b59f12d28159dae3cb89-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"76ead912f9e2b59f12d28159dae3cb89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.419079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.419047 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.419250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.419086 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.419250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.419110 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76ead912f9e2b59f12d28159dae3cb89-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"76ead912f9e2b59f12d28159dae3cb89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.419250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.419156 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76ead912f9e2b59f12d28159dae3cb89-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"76ead912f9e2b59f12d28159dae3cb89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.419250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.419158 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.419250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.419154 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6c6a40b90966d73cbc8b2f5cf79a1c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" (UID: \"d6c6a40b90966d73cbc8b2f5cf79a1c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.823083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.823047 6481 apiserver.go:52] "Watching apiserver" Oct 31 19:17:09.934764 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.934736 6481 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:09.965251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.965221 6481 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 19:17:09.965801 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:09.965780 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-gttls","openshift-console/console-76f6fd84c5-t72cw","openshift-monitoring/monitoring-plugin-77f4f76594-628gq","openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t","openshift-monitoring/node-exporter-gqmbj","kube-system/kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal","openshift-multus/multus-2x85s","openshift-cluster-node-tuning-operator/tuned-p8qf2","openshift-image-registry/node-ca-dc5w2","openshift-insights/insights-runtime-extractor-dx4jp","openshift-monitoring/thanos-querier-849d45575d-5ns87","openshift-ovn-kubernetes/ovnkube-node-ngrxq","openshift-multus/network-metrics-daemon-kjnpm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6","openshift-network-operator/iptables-alerter-76h2j","openshift-ingress-canary/ingress-canary-nlz6k","openshift-monitoring/prometheus-operator-9cd89b74c-4j7np","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal","openshift-console/console-5dcc66d7bc-g2fk7","openshift-dns/node-resolver-vs9x7","openshift-multus/multus-additional-cni-plugins-4s9m5","openshift-network-diagnostics/network-check-target-lf79k","kube-system/konnectivity-agent-mxn7z","openshift-dns/dns-default-2j7sr"] Oct 31 19:17:10.003035 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.002989 6481 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:17:10.003198 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:10.003087 6481 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.us-west-2.compute.internal" Oct 31 19:17:10.015416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015386 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z8492\"" Oct 31 19:17:10.015637 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015609 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:17:10.015777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015642 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.015777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015664 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.015777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015663 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 19:17:10.015777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015713 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qh5p\"" Oct 31 19:17:10.015777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015723 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 19:17:10.015777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015745 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 19:17:10.015777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015762 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015779 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015782 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015847 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015852 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015862 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015881 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d7wnz\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015929 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tgkwh\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.015972 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2kswh\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.016031 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.016105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.016056 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4ftk8\"" Oct 31 19:17:10.032064 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032039 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 19:17:10.032064 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032048 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 19:17:10.032183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032140 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.032183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032168 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.032297 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032283 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.032339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032284 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 19:17:10.032339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032328 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.032402 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032333 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.032448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032432 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.032483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032435 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.032483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032455 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwj7k\"" Oct 31 19:17:10.032483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032467 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 19:17:10.032579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032523 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4b89l\"" Oct 31 19:17:10.032665 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032652 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.032853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032833 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 19:17:10.032945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032895 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.032945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032906 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 19:17:10.032945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032924 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 19:17:10.033084 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.032949 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 19:17:10.033149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.033135 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.033226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.033164 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.033226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.033194 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 19:17:10.033226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.033210 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.033226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.033213 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.033357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.033242 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.037941 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.037926 6481 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 19:17:10.061466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061410 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mcs56\"" Oct 31 19:17:10.061466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061428 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gxs8x\"" Oct 31 19:17:10.061466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061429 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tqdrb\"" Oct 31 19:17:10.061466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061470 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 19:17:10.061714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061489 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.061714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061521 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlrbg\"" Oct 31 19:17:10.061714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061571 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 19:17:10.061714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061600 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:17:10.061714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061629 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 19:17:10.061915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061830 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 19:17:10.061915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061888 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgzhm\"" Oct 31 19:17:10.062030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.061927 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 19:17:10.062075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.062026 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vgfpk\"" Oct 31 19:17:10.062107 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.062075 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zsjk9\"" Oct 31 19:17:10.062159 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.062125 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgqq8\"" Oct 31 19:17:10.115567 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115500 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 19:17:10.115652 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115584 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 19:17:10.115652 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115584 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 19:17:10.115929 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115908 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 19:17:10.116051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115939 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 19:17:10.116051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115962 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 19:17:10.116051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115968 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 19:17:10.116051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.115980 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 19:17:10.116051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116046 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 19:17:10.116305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116073 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 19:17:10.116305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116135 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ptf8h\"" Oct 31 19:17:10.116305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116257 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 19:17:10.116305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116272 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 19:17:10.116460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116356 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 19:17:10.116460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116366 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 19:17:10.116460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116379 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.116555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116541 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 19:17:10.116609 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116558 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 19:17:10.116643 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116567 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q82fc\"" Oct 31 19:17:10.116827 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116806 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 19:17:10.116827 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116816 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bqur2pghjms6i\"" Oct 31 19:17:10.116952 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116832 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 19:17:10.116952 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116832 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bzkl5\"" Oct 31 19:17:10.116952 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116852 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 19:17:10.116952 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116859 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3sj0si341gcar\"" Oct 31 19:17:10.116952 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.116862 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5pcpp\"" Oct 31 19:17:10.117155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.117023 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mvdsq\"" Oct 31 19:17:10.117155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.117042 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 19:17:10.117155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.117105 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 19:17:10.117953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.117936 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 19:17:10.123036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123017 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-device-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.123122 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123067 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84852974-1373-452c-b758-6a5e03bc5ac6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.123122 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123084 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rpc\" (UniqueName: \"kubernetes.io/projected/3c915558-1864-4e8a-8fdf-89439548044c-kube-api-access-86rpc\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.123185 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123176 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-systemd-units\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.123213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123191 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-var-lib-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.123251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123211 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e867aa15-55b5-48e1-8424-f5b176c3bcdb-iptables-alerter-script\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:17:10.123251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123226 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-lib-modules\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.123251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123240 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-root\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.123383 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123298 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtmz\" (UniqueName: \"kubernetes.io/projected/73474282-1f87-4204-969d-2ddcbb5ee758-kube-api-access-vjtmz\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.123383 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123340 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-var-lib-kubelet\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.123473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123448 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-cnibin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.123532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123494 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-bin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.123532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123497 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84852974-1373-452c-b758-6a5e03bc5ac6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.123622 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123536 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e867aa15-55b5-48e1-8424-f5b176c3bcdb-iptables-alerter-script\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:17:10.123622 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123568 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-tmp-dir\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:17:10.123622 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123608 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-kubernetes\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.123754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123627 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-systemd\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.123754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123654 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-client-certs\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.123754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123668 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-tmp-dir\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:17:10.123754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123698 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-netd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.123754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123749 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:10.123983 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123901 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.123983 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123936 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-multus-daemon-config\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.123983 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123948 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-client-certs\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.123983 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123970 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-env-overrides\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.123987 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5aaedfd-3514-492d-96b2-2dc095ff2228-serviceca\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124060 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8pcl\" (UniqueName: \"kubernetes.io/projected/8e993f09-bc04-46f4-9c3a-c77ba0341b81-kube-api-access-f8pcl\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124092 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-node-log\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124121 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-wtmp\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124133 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-multus-daemon-config\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124149 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-env-overrides\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124151 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124159 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.124198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124193 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-dbus\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124211 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124315 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124337 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264ad1b2-15f9-4b71-97ef-96b7f380c650-metrics-client-ca\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124361 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124387 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-tmp\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124415 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66xx\" (UniqueName: \"kubernetes.io/projected/945c708d-ed9a-496a-8cfb-2531e8b9a15a-kube-api-access-f66xx\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124440 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84852974-1373-452c-b758-6a5e03bc5ac6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124455 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-tmp\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124466 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-log-socket\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124495 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.124598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124536 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264ad1b2-15f9-4b71-97ef-96b7f380c650-metrics-client-ca\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124620 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5aaedfd-3514-492d-96b2-2dc095ff2228-serviceca\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124631 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9968b83-2472-46f7-9ca4-664a3b8f89cd-tmp-dir\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124665 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7g2\" (UniqueName: \"kubernetes.io/projected/264ad1b2-15f9-4b71-97ef-96b7f380c650-kube-api-access-tt7g2\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124693 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9968b83-2472-46f7-9ca4-664a3b8f89cd-tmp-dir\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124694 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzq4s\" (UniqueName: \"kubernetes.io/projected/e867aa15-55b5-48e1-8424-f5b176c3bcdb-kube-api-access-qzq4s\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124712 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84852974-1373-452c-b758-6a5e03bc5ac6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124716 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124727 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4rl\" (UniqueName: \"kubernetes.io/projected/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kube-api-access-zz4rl\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124756 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysconfig\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124783 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-conf\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124808 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e993f09-bc04-46f4-9c3a-c77ba0341b81-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124834 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124854 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-netns\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124875 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9968b83-2472-46f7-9ca4-664a3b8f89cd-config-volume\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124893 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-sys-fs\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124908 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-run\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124910 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124930 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.124970 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.125021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125016 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e993f09-bc04-46f4-9c3a-c77ba0341b81-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125097 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125131 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9968b83-2472-46f7-9ca4-664a3b8f89cd-config-volume\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125142 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-systemd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125161 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280701d1-1545-42b0-bb86-39832f46f516-metrics-certs\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125194 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58vt\" (UniqueName: \"kubernetes.io/projected/14c0ff06-8471-4aff-b1a2-ab2450b62cec-kube-api-access-m58vt\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125223 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zssg\" (UniqueName: \"kubernetes.io/projected/280701d1-1545-42b0-bb86-39832f46f516-kube-api-access-4zssg\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125261 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125295 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125310 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-cni-binary-copy\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125326 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125343 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-host\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125426 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84852974-1373-452c-b758-6a5e03bc5ac6-data-volume\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125468 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-system-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125496 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125499 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8vc\" (UniqueName: \"kubernetes.io/projected/f5aaedfd-3514-492d-96b2-2dc095ff2228-kube-api-access-gl8vc\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125519 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125528 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125538 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c915558-1864-4e8a-8fdf-89439548044c-cni-binary-copy\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125544 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-metrics-client-ca\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125570 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84852974-1373-452c-b758-6a5e03bc5ac6-data-volume\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125571 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125613 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-tuned\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125640 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g88\" (UniqueName: \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125668 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xg2\" (UniqueName: \"kubernetes.io/projected/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-kube-api-access-44xg2\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125696 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-hostroot\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125711 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73474282-1f87-4204-969d-2ddcbb5ee758-metrics-client-ca\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125720 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-multus-certs\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125725 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-tuned\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125779 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125796 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.125817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125813 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-metrics-server-audit-profiles\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125850 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125874 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-sys\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125898 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125922 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9976ed46-0a87-46f9-982d-4a4ac667e834-agent-certs\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.125984 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126263 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-metrics-server-audit-profiles\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126269 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5rd\" (UniqueName: \"kubernetes.io/projected/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-kube-api-access-xs5rd\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126320 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126398 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126426 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126448 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df7cc259-0fb7-4f12-ac63-1b798ada6303-audit-log\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126468 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-ovn\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126491 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-script-lib\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126515 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126529 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126541 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-system-cni-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126560 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126573 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhzf\" (UniqueName: \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126622 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9976ed46-0a87-46f9-982d-4a4ac667e834-agent-certs\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126635 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-socket-dir-parent\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126697 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df7cc259-0fb7-4f12-ac63-1b798ada6303-audit-log\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126697 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df7cc259-0fb7-4f12-ac63-1b798ada6303-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126688 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-socket-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126742 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-tls\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126786 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126794 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126804 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-etc-kubernetes\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126809 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb981019-7d15-475e-ac02-ef33f915867a-original-pull-secret\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126871 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126901 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-netns\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126929 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-config\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126961 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-grpc-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126960 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-script-lib\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.126977 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-secret-metrics-server-tls\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127030 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cnibin\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127053 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.127099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127074 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127139 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-modprobe-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127145 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovnkube-config\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127188 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127219 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-grpc-tls\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127223 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127231 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127262 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8n7\" (UniqueName: \"kubernetes.io/projected/df7cc259-0fb7-4f12-ac63-1b798ada6303-kube-api-access-7r8n7\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127288 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2kr\" (UniqueName: \"kubernetes.io/projected/4fc92bb3-b6ec-4407-b9dd-6082981560e6-kube-api-access-sd2kr\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127341 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127341 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-textfile\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127398 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e993f09-bc04-46f4-9c3a-c77ba0341b81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127440 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127453 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-textfile\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127489 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127579 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127621 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127672 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/264ad1b2-15f9-4b71-97ef-96b7f380c650-tmp\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127679 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127758 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-etc-selinux\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127775 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127820 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/264ad1b2-15f9-4b71-97ef-96b7f380c650-tmp\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127819 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127844 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9flg\" (UniqueName: \"kubernetes.io/projected/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-kube-api-access-x9flg\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127860 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127883 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9976ed46-0a87-46f9-982d-4a4ac667e834-konnectivity-ca\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127892 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc92bb3-b6ec-4407-b9dd-6082981560e6-ovn-node-metrics-cert\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127935 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14c0ff06-8471-4aff-b1a2-ab2450b62cec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.127987 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-kubelet\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128094 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-conf-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128122 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-bin\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128138 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128144 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9976ed46-0a87-46f9-982d-4a4ac667e834-konnectivity-ca\") pod \"konnectivity-agent-mxn7z\" (UID: \"9976ed46-0a87-46f9-982d-4a4ac667e834\") " pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128154 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-k8s-cni-cncf-io\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128183 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-kubelet\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128208 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-tls\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128239 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-hosts-file\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128272 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-registration-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128315 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128343 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-multus\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128394 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-tls\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128411 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-slash\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.128427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128447 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-sys\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128479 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128498 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128525 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-etc-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128558 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5aaedfd-3514-492d-96b2-2dc095ff2228-host\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128587 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9968b83-2472-46f7-9ca4-664a3b8f89cd-metrics-tls\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128623 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktr6\" (UniqueName: \"kubernetes.io/projected/e9968b83-2472-46f7-9ca4-664a3b8f89cd-kube-api-access-gktr6\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128720 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128738 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e867aa15-55b5-48e1-8424-f5b176c3bcdb-host-slash\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128770 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128793 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-client-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128816 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9968b83-2472-46f7-9ca4-664a3b8f89cd-metrics-tls\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128832 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128930 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128946 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-cert\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128962 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84852974-1373-452c-b758-6a5e03bc5ac6-crio-socket\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128979 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr72\" (UniqueName: \"kubernetes.io/projected/84852974-1373-452c-b758-6a5e03bc5ac6-kube-api-access-mxr72\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.129011 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.128945 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.129068 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-kubelet-config\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.129123 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df7cc259-0fb7-4f12-ac63-1b798ada6303-client-ca-bundle\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.129184 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.129277 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.129351 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264ad1b2-15f9-4b71-97ef-96b7f380c650-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.129949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.129396 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-cert\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:17:10.177233 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.177198 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rpc\" (UniqueName: \"kubernetes.io/projected/3c915558-1864-4e8a-8fdf-89439548044c-kube-api-access-86rpc\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.178134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.178106 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2zr\" (UniqueName: \"kubernetes.io/projected/efc46de0-c9f5-4659-9713-631a24271f0d-kube-api-access-6n2zr\") pod \"network-check-target-lf79k\" (UID: \"efc46de0-c9f5-4659-9713-631a24271f0d\") " pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:10.178258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.178153 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzq4s\" (UniqueName: \"kubernetes.io/projected/e867aa15-55b5-48e1-8424-f5b176c3bcdb-kube-api-access-qzq4s\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:17:10.178369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.178351 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtmz\" (UniqueName: \"kubernetes.io/projected/73474282-1f87-4204-969d-2ddcbb5ee758-kube-api-access-vjtmz\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.178467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.178447 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8pcl\" (UniqueName: \"kubernetes.io/projected/8e993f09-bc04-46f4-9c3a-c77ba0341b81-kube-api-access-f8pcl\") pod \"prometheus-operator-9cd89b74c-4j7np\" (UID: \"8e993f09-bc04-46f4-9c3a-c77ba0341b81\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4j7np" Oct 31 19:17:10.180852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.180831 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xg2\" (UniqueName: \"kubernetes.io/projected/06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc-kube-api-access-44xg2\") pod \"monitoring-plugin-77f4f76594-628gq\" (UID: \"06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:17:10.180852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.180841 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zssg\" (UniqueName: \"kubernetes.io/projected/280701d1-1545-42b0-bb86-39832f46f516-kube-api-access-4zssg\") pod \"network-metrics-daemon-kjnpm\" (UID: \"280701d1-1545-42b0-bb86-39832f46f516\") " pod="openshift-multus/network-metrics-daemon-kjnpm" Oct 31 19:17:10.180852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.180844 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4rl\" (UniqueName: \"kubernetes.io/projected/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kube-api-access-zz4rl\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.181060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.180940 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g88\" (UniqueName: \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\") pod \"console-76f6fd84c5-t72cw\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:10.181158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.181142 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhzf\" (UniqueName: \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\") pod \"console-5dcc66d7bc-g2fk7\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:10.182179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.182157 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8vc\" (UniqueName: \"kubernetes.io/projected/f5aaedfd-3514-492d-96b2-2dc095ff2228-kube-api-access-gl8vc\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:17:10.182295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.182277 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8n7\" (UniqueName: \"kubernetes.io/projected/df7cc259-0fb7-4f12-ac63-1b798ada6303-kube-api-access-7r8n7\") pod \"metrics-server-75bb7ff6dd-6qm4t\" (UID: \"df7cc259-0fb7-4f12-ac63-1b798ada6303\") " pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:10.182612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.182588 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9flg\" (UniqueName: \"kubernetes.io/projected/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-kube-api-access-x9flg\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:17:10.182612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.182603 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5rd\" (UniqueName: \"kubernetes.io/projected/61f4f0c2-7d69-487c-bfd0-cc4e5da89716-kube-api-access-xs5rd\") pod \"ingress-canary-nlz6k\" (UID: \"61f4f0c2-7d69-487c-bfd0-cc4e5da89716\") " pod="openshift-ingress-canary/ingress-canary-nlz6k" Oct 31 19:17:10.182787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.182771 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7g2\" (UniqueName: \"kubernetes.io/projected/264ad1b2-15f9-4b71-97ef-96b7f380c650-kube-api-access-tt7g2\") pod \"thanos-querier-849d45575d-5ns87\" (UID: \"264ad1b2-15f9-4b71-97ef-96b7f380c650\") " pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:10.183353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.183337 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58vt\" (UniqueName: \"kubernetes.io/projected/14c0ff06-8471-4aff-b1a2-ab2450b62cec-kube-api-access-m58vt\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.183419 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.183406 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2kr\" (UniqueName: \"kubernetes.io/projected/4fc92bb3-b6ec-4407-b9dd-6082981560e6-kube-api-access-sd2kr\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.183767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.183748 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66xx\" (UniqueName: \"kubernetes.io/projected/945c708d-ed9a-496a-8cfb-2531e8b9a15a-kube-api-access-f66xx\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.184006 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.183981 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr72\" (UniqueName: \"kubernetes.io/projected/84852974-1373-452c-b758-6a5e03bc5ac6-kube-api-access-mxr72\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.184391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.184376 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktr6\" (UniqueName: \"kubernetes.io/projected/e9968b83-2472-46f7-9ca4-664a3b8f89cd-kube-api-access-gktr6\") pod \"dns-default-2j7sr\" (UID: \"e9968b83-2472-46f7-9ca4-664a3b8f89cd\") " pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:10.229819 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229789 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-netns\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.229819 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229822 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-sys-fs\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229836 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-run\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229851 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229872 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229927 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229930 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-run\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229927 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229929 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-sys-fs\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229945 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-systemd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.229974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229961 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-netns\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229977 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-systemd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.229982 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-host\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230037 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-system-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230041 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-host\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230064 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230082 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-hostroot\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230096 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-multus-certs\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230102 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-system-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230111 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-sys\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230130 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-ovn\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230142 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-hostroot\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230156 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-multus-certs\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230170 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-run-ovn\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230175 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-system-cni-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230179 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230183 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-sys\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230198 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-socket-dir-parent\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230213 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-socket-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230230 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-etc-kubernetes\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.230272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230271 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-etc-kubernetes\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230284 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-socket-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230313 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-netns\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230343 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cnibin\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230316 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-socket-dir-parent\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230364 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230343 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-system-cni-dir\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230390 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-modprobe-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230400 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-netns\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230416 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14c0ff06-8471-4aff-b1a2-ab2450b62cec-cnibin\") pod \"multus-additional-cni-plugins-4s9m5\" (UID: \"14c0ff06-8471-4aff-b1a2-ab2450b62cec\") " pod="openshift-multus/multus-additional-cni-plugins-4s9m5" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230422 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230428 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-etc-selinux\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230458 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-etc-selinux\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230480 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-kubelet\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230508 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-conf-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230532 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-conf-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230534 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-bin\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230482 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-modprobe-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230511 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-kubelet\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230557 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230581 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-k8s-cni-cncf-io\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230604 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-kubelet\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230630 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-hosts-file\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230636 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-run-k8s-cni-cncf-io\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230645 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-bin\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230652 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-multus-cni-dir\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230659 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-kubelet\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230670 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-registration-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230689 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6-hosts-file\") pod \"node-resolver-vs9x7\" (UID: \"05d1a878-d2e1-4a6f-aa0b-ec6cd892b5a6\") " pod="openshift-dns/node-resolver-vs9x7" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230704 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-multus\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230729 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-slash\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230743 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-multus\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230744 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-registration-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230751 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-sys\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230772 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-slash\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.231081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.230839 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-sys\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231133 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-etc-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231193 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5aaedfd-3514-492d-96b2-2dc095ff2228-host\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231203 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-etc-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231226 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e867aa15-55b5-48e1-8424-f5b176c3bcdb-host-slash\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231262 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5aaedfd-3514-492d-96b2-2dc095ff2228-host\") pod \"node-ca-dc5w2\" (UID: \"f5aaedfd-3514-492d-96b2-2dc095ff2228\") " pod="openshift-image-registry/node-ca-dc5w2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231297 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e867aa15-55b5-48e1-8424-f5b176c3bcdb-host-slash\") pod \"iptables-alerter-76h2j\" (UID: \"e867aa15-55b5-48e1-8424-f5b176c3bcdb\") " pod="openshift-network-operator/iptables-alerter-76h2j" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231371 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231431 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84852974-1373-452c-b758-6a5e03bc5ac6-crio-socket\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231465 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231507 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-kubelet-config\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231538 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-device-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231588 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-systemd-units\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231620 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-var-lib-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231670 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-lib-modules\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231696 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-root\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231743 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-var-lib-kubelet\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231775 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-cnibin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231820 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-bin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231849 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-kubernetes\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231896 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-systemd\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231930 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-netd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.231981 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-node-log\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232039 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-wtmp\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232067 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-dbus\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232121 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-log-socket\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232171 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysconfig\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232204 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-conf\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232440 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-conf\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232525 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.232672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232599 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84852974-1373-452c-b758-6a5e03bc5ac6-crio-socket\") pod \"insights-runtime-extractor-dx4jp\" (UID: \"84852974-1373-452c-b758-6a5e03bc5ac6\") " pod="openshift-insights/insights-runtime-extractor-dx4jp" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232761 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysctl-d\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232848 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-kubelet-config\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232923 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f90009c-b0e7-4288-ac5f-2a5fdc462b18-device-dir\") pod \"aws-ebs-csi-driver-node-2q2v6\" (UID: \"0f90009c-b0e7-4288-ac5f-2a5fdc462b18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2q2v6" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.232988 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-systemd-units\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233060 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-var-lib-openvswitch\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233201 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-lib-modules\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233276 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-root\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233351 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-var-lib-kubelet\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233428 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-cnibin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233492 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c915558-1864-4e8a-8fdf-89439548044c-host-var-lib-cni-bin\") pod \"multus-2x85s\" (UID: \"3c915558-1864-4e8a-8fdf-89439548044c\") " pod="openshift-multus/multus-2x85s" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233571 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-kubernetes\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233647 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-systemd\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233714 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-host-cni-netd\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233763 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-node-log\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233837 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/73474282-1f87-4204-969d-2ddcbb5ee758-node-exporter-wtmp\") pod \"node-exporter-gqmbj\" (UID: \"73474282-1f87-4204-969d-2ddcbb5ee758\") " pod="openshift-monitoring/node-exporter-gqmbj" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.233949 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb981019-7d15-475e-ac02-ef33f915867a-dbus\") pod \"global-pull-secret-syncer-gttls\" (UID: \"fb981019-7d15-475e-ac02-ef33f915867a\") " pod="kube-system/global-pull-secret-syncer-gttls" Oct 31 19:17:10.234078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.234029 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc92bb3-b6ec-4407-b9dd-6082981560e6-log-socket\") pod \"ovnkube-node-ngrxq\" (UID: \"4fc92bb3-b6ec-4407-b9dd-6082981560e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:10.234795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.234115 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/945c708d-ed9a-496a-8cfb-2531e8b9a15a-etc-sysconfig\") pod \"tuned-p8qf2\" (UID: \"945c708d-ed9a-496a-8cfb-2531e8b9a15a\") " pod="openshift-cluster-node-tuning-operator/tuned-p8qf2" Oct 31 19:17:10.535008 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:10.534965 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-849d45575d-5ns87" Oct 31 19:17:11.137009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:11.136957 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2j7sr" Oct 31 19:17:11.629652 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:11.629607 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:14.148152 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:14.148124 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:17:14.301900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:14.301869 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f6fd84c5-t72cw"] Oct 31 19:17:14.396509 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:14.396478 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:18.373216 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:18.373187 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:18.598039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:18.597987 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-628gq" Oct 31 19:17:19.262655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:19.262587 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:19.262843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:19.262826 6481 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:17:27.323954 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:27.323913 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:17:30.982443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:30.982413 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngrxq" Oct 31 19:17:32.011867 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:32.011839 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75bb7ff6dd-6qm4t" Oct 31 19:17:34.212474 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:34.212438 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxn7z" Oct 31 19:17:39.323118 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.323055 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76f6fd84c5-t72cw" podUID="d4d70a7b-5d24-4d0f-909e-56b6b8273060" containerName="console" containerID="cri-o://8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d" gracePeriod=15 Oct 31 19:17:39.614447 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.614421 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:39.749899 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.749870 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\") pod \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " Oct 31 19:17:39.750129 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.749936 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\") pod \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " Oct 31 19:17:39.750129 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.749977 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\") pod \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " Oct 31 19:17:39.750129 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750020 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\") pod \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " Oct 31 19:17:39.750129 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750053 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\") pod \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " Oct 31 19:17:39.750129 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750102 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\") pod \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " Oct 31 19:17:39.750408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750150 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\" (UniqueName: \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\") pod \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\" (UID: \"d4d70a7b-5d24-4d0f-909e-56b6b8273060\") " Oct 31 19:17:39.750408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750311 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp" pod "d4d70a7b-5d24-4d0f-909e-56b6b8273060" (UID: "d4d70a7b-5d24-4d0f-909e-56b6b8273060"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:17:39.750408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750383 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config" pod "d4d70a7b-5d24-4d0f-909e-56b6b8273060" (UID: "d4d70a7b-5d24-4d0f-909e-56b6b8273060"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:39.750563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750428 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca" pod "d4d70a7b-5d24-4d0f-909e-56b6b8273060" (UID: "d4d70a7b-5d24-4d0f-909e-56b6b8273060"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:39.750771 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.750745 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert" pod "d4d70a7b-5d24-4d0f-909e-56b6b8273060" (UID: "d4d70a7b-5d24-4d0f-909e-56b6b8273060"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:17:39.752319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.752294 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config" pod "d4d70a7b-5d24-4d0f-909e-56b6b8273060" (UID: "d4d70a7b-5d24-4d0f-909e-56b6b8273060"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:39.752416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.752378 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88" pod "d4d70a7b-5d24-4d0f-909e-56b6b8273060" (UID: "d4d70a7b-5d24-4d0f-909e-56b6b8273060"). InnerVolumeSpecName "kube-api-access-74g88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:17:39.752416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.752387 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert" pod "d4d70a7b-5d24-4d0f-909e-56b6b8273060" (UID: "d4d70a7b-5d24-4d0f-909e-56b6b8273060"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:17:39.851574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.851513 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74g88\" (UniqueName: \"kubernetes.io/projected/d4d70a7b-5d24-4d0f-909e-56b6b8273060-kube-api-access-74g88\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:39.851574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.851538 6481 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-oauth-config\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:39.851574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.851550 6481 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-config\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:39.851574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.851558 6481 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d70a7b-5d24-4d0f-909e-56b6b8273060-tmp\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:39.851574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.851567 6481 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:39.851574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.851576 6481 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4d70a7b-5d24-4d0f-909e-56b6b8273060-console-serving-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:39.851851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:39.851584 6481 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4d70a7b-5d24-4d0f-909e-56b6b8273060-oauth-serving-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:17:40.035559 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.035521 6481 generic.go:356] "Generic (PLEG): container finished" podID="d4d70a7b-5d24-4d0f-909e-56b6b8273060" containerID="8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d" exitCode=0 Oct 31 19:17:40.035723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.035571 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f6fd84c5-t72cw" Oct 31 19:17:40.035723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.035610 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6fd84c5-t72cw" event={"ID":"d4d70a7b-5d24-4d0f-909e-56b6b8273060","Type":"ContainerDied","Data":"8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d"} Oct 31 19:17:40.035723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.035665 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f6fd84c5-t72cw" event={"ID":"d4d70a7b-5d24-4d0f-909e-56b6b8273060","Type":"ContainerDied","Data":"31cc8052b06bb4208d19296dd841cd7a00c0ca33325b68639aceb46c1b541380"} Oct 31 19:17:40.035723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.035687 6481 scope.go:122] "RemoveContainer" containerID="8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d" Oct 31 19:17:40.050411 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.050392 6481 scope.go:122] "RemoveContainer" containerID="8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d" Oct 31 19:17:40.050683 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:40.050663 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d\": container with ID starting with 8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d not found: ID does not exist" containerID="8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d" Oct 31 19:17:40.050737 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.050692 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d"} err="failed to get container status \"8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d\": rpc error: code = NotFound desc = could not find container \"8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d\": container with ID starting with 8ff43786db69d30c9530f16dfafcc9251fb6b92ffe151d59f1da880eb518f52d not found: ID does not exist" Oct 31 19:17:40.185516 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.185441 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f6fd84c5-t72cw"] Oct 31 19:17:40.188736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.188709 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76f6fd84c5-t72cw"] Oct 31 19:17:40.877499 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:40.877460 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d70a7b-5d24-4d0f-909e-56b6b8273060" path="/var/lib/kubelet/pods/d4d70a7b-5d24-4d0f-909e-56b6b8273060/volumes" Oct 31 19:17:41.632587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:41.632551 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lf79k" Oct 31 19:17:44.375291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.375256 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69454877bc-frmmb"] Oct 31 19:17:44.379443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.376210 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d4d70a7b-5d24-4d0f-909e-56b6b8273060" containerName="console" Oct 31 19:17:44.379443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.376239 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d70a7b-5d24-4d0f-909e-56b6b8273060" containerName="console" Oct 31 19:17:44.379443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.376497 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d70a7b-5d24-4d0f-909e-56b6b8273060" containerName="console" Oct 31 19:17:44.380512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.380482 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.425924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.425890 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69454877bc-frmmb"] Oct 31 19:17:44.471203 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.471173 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 19:17:44.471203 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.471171 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 19:17:44.471445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.471176 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 19:17:44.471445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.471278 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 19:17:44.471445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.471303 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nqpq8\"" Oct 31 19:17:44.471445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.471367 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 19:17:44.493304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493275 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-telemeter-client-tls\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.493439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493320 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.493439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493346 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-metrics-client-ca\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.493518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493496 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-secret-telemeter-client\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.493572 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493558 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-serving-certs-ca-bundle\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.493606 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493588 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-federate-client-tls\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.493646 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493608 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.493646 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.493624 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gld2r\" (UniqueName: \"kubernetes.io/projected/edfc0931-1a29-4aaf-b2e2-954b25a473e5-kube-api-access-gld2r\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.521367 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.521340 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 19:17:44.594882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.594844 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.594882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.594883 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-metrics-client-ca\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595167 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.594933 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-secret-telemeter-client\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595167 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.594984 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-serving-certs-ca-bundle\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595167 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.595032 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-federate-client-tls\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595167 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.595059 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595167 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.595081 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gld2r\" (UniqueName: \"kubernetes.io/projected/edfc0931-1a29-4aaf-b2e2-954b25a473e5-kube-api-access-gld2r\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595167 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.595119 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-telemeter-client-tls\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.595329 6481 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 19:17:44.595845 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.595815 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-metrics-client-ca\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.595960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.595852 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-serving-certs-ca-bundle\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.596078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.596060 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edfc0931-1a29-4aaf-b2e2-954b25a473e5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.597626 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.597598 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.597744 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.597687 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-telemeter-client-tls\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.598114 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.598094 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-secret-telemeter-client\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.598160 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.598109 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/edfc0931-1a29-4aaf-b2e2-954b25a473e5-federate-client-tls\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.647980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.647918 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gld2r\" (UniqueName: \"kubernetes.io/projected/edfc0931-1a29-4aaf-b2e2-954b25a473e5-kube-api-access-gld2r\") pod \"telemeter-client-69454877bc-frmmb\" (UID: \"edfc0931-1a29-4aaf-b2e2-954b25a473e5\") " pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:44.692736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:44.692705 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" Oct 31 19:17:45.155476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:45.155447 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69454877bc-frmmb"] Oct 31 19:17:45.158726 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:45.158695 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfc0931_1a29_4aaf_b2e2_954b25a473e5.slice/crio-0a0cddfca147a544628b16e74aed08fdf8f23e4806453fc71b40684efd1676c1 WatchSource:0}: Error finding container 0a0cddfca147a544628b16e74aed08fdf8f23e4806453fc71b40684efd1676c1: Status 404 returned error can't find the container with id 0a0cddfca147a544628b16e74aed08fdf8f23e4806453fc71b40684efd1676c1 Oct 31 19:17:45.160676 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:45.160654 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:17:46.057465 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:46.057426 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" event={"ID":"edfc0931-1a29-4aaf-b2e2-954b25a473e5","Type":"ContainerStarted","Data":"0a0cddfca147a544628b16e74aed08fdf8f23e4806453fc71b40684efd1676c1"} Oct 31 19:17:48.067753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:48.067714 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" event={"ID":"edfc0931-1a29-4aaf-b2e2-954b25a473e5","Type":"ContainerStarted","Data":"d3ee4a7731140f36d7c34b4ab1745b8778d8b8b37cf39eb68822b20a020fd17b"} Oct 31 19:17:49.073788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.073735 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" event={"ID":"edfc0931-1a29-4aaf-b2e2-954b25a473e5","Type":"ContainerStarted","Data":"42f2b6c51cc815f92a1d92d3af996f7c4aedbf39b14e0412d495048db7bf404c"} Oct 31 19:17:49.074312 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.073804 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" event={"ID":"edfc0931-1a29-4aaf-b2e2-954b25a473e5","Type":"ContainerStarted","Data":"f1240d3b22d274158d6bdc8fdd695d9bbd921c45d78f4b02f18a70334649764d"} Oct 31 19:17:49.337418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.337358 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69454877bc-frmmb" podStartSLOduration=1.626751541 podStartE2EDuration="5.337346634s" totalImagesPullingTime="3.710595093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:49.337129071 +0000 UTC m=+40.577352199" watchObservedRunningTime="2026-10-31 19:17:49.337346634 +0000 UTC m=+40.577569754" Oct 31 19:17:49.811160 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.811121 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7674df7bfb-55bm8"] Oct 31 19:17:49.816418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.816383 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.869874 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.869839 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7674df7bfb-55bm8"] Oct 31 19:17:49.939736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.939706 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-service-ca\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.939736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.939742 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-oauth-serving-cert\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.940039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.939766 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-trusted-ca-bundle\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.940039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.939829 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-serving-cert\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.940039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.939874 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-config\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.940039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.939965 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-oauth-config\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.940039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.940035 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mkh\" (UniqueName: \"kubernetes.io/projected/7f3e3af3-8266-4e8f-97c4-e2681bad078b-kube-api-access-z5mkh\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:49.940246 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:49.940069 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3af3-8266-4e8f-97c4-e2681bad078b-tmp\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041245 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041205 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mkh\" (UniqueName: \"kubernetes.io/projected/7f3e3af3-8266-4e8f-97c4-e2681bad078b-kube-api-access-z5mkh\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041245 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041248 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3af3-8266-4e8f-97c4-e2681bad078b-tmp\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041272 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-service-ca\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041292 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-oauth-serving-cert\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041480 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-trusted-ca-bundle\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041527 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-serving-cert\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041569 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-config\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041632 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-oauth-config\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.041738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.041681 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3af3-8266-4e8f-97c4-e2681bad078b-tmp\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.042101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.042073 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-service-ca\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.042227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.042177 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-oauth-serving-cert\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.042304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.042276 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-config\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.042440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.042418 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-trusted-ca-bundle\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.044070 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.044048 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-oauth-config\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.044149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.044135 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-serving-cert\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.095500 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.095433 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mkh\" (UniqueName: \"kubernetes.io/projected/7f3e3af3-8266-4e8f-97c4-e2681bad078b-kube-api-access-z5mkh\") pod \"console-7674df7bfb-55bm8\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.127265 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.127236 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:17:50.494827 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:50.494794 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7674df7bfb-55bm8"] Oct 31 19:17:50.498476 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:17:50.498450 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3e3af3_8266_4e8f_97c4_e2681bad078b.slice/crio-daf11d9f375ef07a83e4e2f78c3394f37c14c9d7a11434089bfd2a05a51ef0e9 WatchSource:0}: Error finding container daf11d9f375ef07a83e4e2f78c3394f37c14c9d7a11434089bfd2a05a51ef0e9: Status 404 returned error can't find the container with id daf11d9f375ef07a83e4e2f78c3394f37c14c9d7a11434089bfd2a05a51ef0e9 Oct 31 19:17:51.086471 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:51.086436 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7674df7bfb-55bm8" event={"ID":"7f3e3af3-8266-4e8f-97c4-e2681bad078b","Type":"ContainerStarted","Data":"8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114"} Oct 31 19:17:51.086643 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:51.086486 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7674df7bfb-55bm8" event={"ID":"7f3e3af3-8266-4e8f-97c4-e2681bad078b","Type":"ContainerStarted","Data":"daf11d9f375ef07a83e4e2f78c3394f37c14c9d7a11434089bfd2a05a51ef0e9"} Oct 31 19:17:51.200182 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:17:51.200126 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7674df7bfb-55bm8" podStartSLOduration=2.200111069 podStartE2EDuration="2.200111069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:17:51.199187259 +0000 UTC m=+42.439410388" watchObservedRunningTime="2026-10-31 19:17:51.200111069 +0000 UTC m=+42.440334192" Oct 31 19:17:57.323095 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:17:57.323058 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:18:00.128124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:00.128091 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:18:00.128548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:00.128170 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:18:00.132779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:00.132756 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:18:01.121643 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:01.121615 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:18:01.276414 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:01.276372 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcc66d7bc-g2fk7"] Oct 31 19:18:26.295285 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.295240 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5dcc66d7bc-g2fk7" podUID="931e6634-fb0c-498e-b225-59a8deb17655" containerName="console" containerID="cri-o://021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997" gracePeriod=15 Oct 31 19:18:26.540794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.540770 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:18:26.612663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.612577 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.612663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.612634 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.612870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.612707 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\" (UniqueName: \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.612870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.612737 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.612870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.612778 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.612870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.612816 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.613083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.613031 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:18:26.613083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.613070 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:18:26.613242 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.613212 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:18:26.613331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.613301 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:18:26.614841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.614819 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:18:26.614926 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.614899 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:18:26.713291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713260 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.713446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713297 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\" (UniqueName: \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\") pod \"931e6634-fb0c-498e-b225-59a8deb17655\" (UID: \"931e6634-fb0c-498e-b225-59a8deb17655\") " Oct 31 19:18:26.713513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713467 6481 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/931e6634-fb0c-498e-b225-59a8deb17655-tmp\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:26.713513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713490 6481 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-trusted-ca-bundle\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:26.713513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713507 6481 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-oauth-config\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:26.713664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713521 6481 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:26.713664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713536 6481 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-oauth-serving-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:26.713664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713552 6481 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/931e6634-fb0c-498e-b225-59a8deb17655-console-serving-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:26.713664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.713623 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:18:26.715355 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.715332 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf" pod "931e6634-fb0c-498e-b225-59a8deb17655" (UID: "931e6634-fb0c-498e-b225-59a8deb17655"). InnerVolumeSpecName "kube-api-access-6vhzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:18:26.813888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.813848 6481 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/931e6634-fb0c-498e-b225-59a8deb17655-console-config\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:26.813888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:26.813885 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhzf\" (UniqueName: \"kubernetes.io/projected/931e6634-fb0c-498e-b225-59a8deb17655-kube-api-access-6vhzf\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:18:27.202966 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.202882 6481 generic.go:356] "Generic (PLEG): container finished" podID="931e6634-fb0c-498e-b225-59a8deb17655" containerID="021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997" exitCode=0 Oct 31 19:18:27.202966 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.202940 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc66d7bc-g2fk7" Oct 31 19:18:27.203206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.202942 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc66d7bc-g2fk7" event={"ID":"931e6634-fb0c-498e-b225-59a8deb17655","Type":"ContainerDied","Data":"021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997"} Oct 31 19:18:27.203206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.203073 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc66d7bc-g2fk7" event={"ID":"931e6634-fb0c-498e-b225-59a8deb17655","Type":"ContainerDied","Data":"283b9d33df5fef77ce1cdc347f0e3df9bda175e8a6d82b831e4e454cb3ea9529"} Oct 31 19:18:27.203206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.203108 6481 scope.go:122] "RemoveContainer" containerID="021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997" Oct 31 19:18:27.210707 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.210689 6481 scope.go:122] "RemoveContainer" containerID="021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997" Oct 31 19:18:27.211010 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:18:27.210972 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997\": container with ID starting with 021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997 not found: ID does not exist" containerID="021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997" Oct 31 19:18:27.211082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.211016 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997"} err="failed to get container status \"021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997\": rpc error: code = NotFound desc = could not find container \"021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997\": container with ID starting with 021499f118397eef93538b992800709c3e5b13a0211ef2a12e5a0e6d935dd997 not found: ID does not exist" Oct 31 19:18:27.324674 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:18:27.324637 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:18:27.359845 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.359814 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcc66d7bc-g2fk7"] Oct 31 19:18:27.406388 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:27.406351 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dcc66d7bc-g2fk7"] Oct 31 19:18:28.876984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:18:28.876942 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931e6634-fb0c-498e-b225-59a8deb17655" path="/var/lib/kubelet/pods/931e6634-fb0c-498e-b225-59a8deb17655/volumes" Oct 31 19:18:57.323450 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:18:57.323372 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:19:01.045239 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.045211 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd667d795-5pxzj"] Oct 31 19:19:01.045631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.045476 6481 cpu_manager.go:414] "containerMap: removing container" podUID="931e6634-fb0c-498e-b225-59a8deb17655" containerName="console" Oct 31 19:19:01.045631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.045486 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931e6634-fb0c-498e-b225-59a8deb17655" containerName="console" Oct 31 19:19:01.045631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.045536 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="931e6634-fb0c-498e-b225-59a8deb17655" containerName="console" Oct 31 19:19:01.048121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.048105 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.142890 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.142809 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd667d795-5pxzj"] Oct 31 19:19:01.152083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152055 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-service-ca\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.152204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152148 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-console-config\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.152204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152184 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef39d7d-6216-4f9d-b439-26613c92d366-tmp\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.152279 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152218 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-trusted-ca-bundle\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.152279 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152252 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-oauth-serving-cert\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.152349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152298 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p2p\" (UniqueName: \"kubernetes.io/projected/3ef39d7d-6216-4f9d-b439-26613c92d366-kube-api-access-p9p2p\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.152349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152336 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef39d7d-6216-4f9d-b439-26613c92d366-console-oauth-config\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.152432 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.152356 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef39d7d-6216-4f9d-b439-26613c92d366-console-serving-cert\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.253579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253541 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef39d7d-6216-4f9d-b439-26613c92d366-console-serving-cert\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.253579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253586 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-service-ca\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.253821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253617 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-console-config\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.253821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253642 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef39d7d-6216-4f9d-b439-26613c92d366-tmp\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.253821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253668 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-trusted-ca-bundle\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.253821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253696 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-oauth-serving-cert\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.253821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253802 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p2p\" (UniqueName: \"kubernetes.io/projected/3ef39d7d-6216-4f9d-b439-26613c92d366-kube-api-access-p9p2p\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.254075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.253836 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef39d7d-6216-4f9d-b439-26613c92d366-console-oauth-config\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.254284 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.254258 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ef39d7d-6216-4f9d-b439-26613c92d366-tmp\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.254510 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.254490 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-oauth-serving-cert\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.254510 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.254496 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-console-config\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.254646 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.254624 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-service-ca\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.254690 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.254626 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ef39d7d-6216-4f9d-b439-26613c92d366-trusted-ca-bundle\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.256299 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.256278 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ef39d7d-6216-4f9d-b439-26613c92d366-console-oauth-config\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.256414 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.256396 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ef39d7d-6216-4f9d-b439-26613c92d366-console-serving-cert\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.306860 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.306794 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p2p\" (UniqueName: \"kubernetes.io/projected/3ef39d7d-6216-4f9d-b439-26613c92d366-kube-api-access-p9p2p\") pod \"console-6cd667d795-5pxzj\" (UID: \"3ef39d7d-6216-4f9d-b439-26613c92d366\") " pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.359054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.359025 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:01.766398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:01.766366 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd667d795-5pxzj"] Oct 31 19:19:01.770373 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:19:01.770346 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef39d7d_6216_4f9d_b439_26613c92d366.slice/crio-fc1967ad74c0955972f57223d9cc35725fc834466ccd7f8223d05e1ce35527be WatchSource:0}: Error finding container fc1967ad74c0955972f57223d9cc35725fc834466ccd7f8223d05e1ce35527be: Status 404 returned error can't find the container with id fc1967ad74c0955972f57223d9cc35725fc834466ccd7f8223d05e1ce35527be Oct 31 19:19:02.325252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:02.325214 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd667d795-5pxzj" event={"ID":"3ef39d7d-6216-4f9d-b439-26613c92d366","Type":"ContainerStarted","Data":"859849280e2dd5560e5081b770c8a10e8ba1404342d0d51bfa4e15362219191e"} Oct 31 19:19:02.325728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:02.325264 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd667d795-5pxzj" event={"ID":"3ef39d7d-6216-4f9d-b439-26613c92d366","Type":"ContainerStarted","Data":"fc1967ad74c0955972f57223d9cc35725fc834466ccd7f8223d05e1ce35527be"} Oct 31 19:19:02.433393 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:02.433334 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cd667d795-5pxzj" podStartSLOduration=1.4333192719999999 podStartE2EDuration="1.433319272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:02.433308137 +0000 UTC m=+113.673531278" watchObservedRunningTime="2026-10-31 19:19:02.433319272 +0000 UTC m=+113.673542403" Oct 31 19:19:11.359277 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:11.359244 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:11.359698 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:11.359301 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:11.363911 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:11.363887 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:12.361087 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:12.361061 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cd667d795-5pxzj" Oct 31 19:19:12.523310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:12.523278 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7674df7bfb-55bm8"] Oct 31 19:19:27.324020 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:19:27.323963 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:19:37.542198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.542131 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7674df7bfb-55bm8" podUID="7f3e3af3-8266-4e8f-97c4-e2681bad078b" containerName="console" containerID="cri-o://8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114" gracePeriod=15 Oct 31 19:19:37.779578 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.779549 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:19:37.950545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950454 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-oauth-serving-cert\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950504 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3e3af3-8266-4e8f-97c4-e2681bad078b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3af3-8266-4e8f-97c4-e2681bad078b-tmp\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950528 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-serving-cert\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950547 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-config\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-config\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950597 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3e3af3-8266-4e8f-97c4-e2681bad078b-kube-api-access-z5mkh\" (UniqueName: \"kubernetes.io/projected/7f3e3af3-8266-4e8f-97c4-e2681bad078b-kube-api-access-z5mkh\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950639 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-oauth-config\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950680 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-trusted-ca-bundle\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950722 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-service-ca\") pod \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\" (UID: \"7f3e3af3-8266-4e8f-97c4-e2681bad078b\") " Oct 31 19:19:37.950904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950795 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3e3af3-8266-4e8f-97c4-e2681bad078b-tmp" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:19:37.951161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.950915 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-oauth-serving-cert" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:19:37.951161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.951054 6481 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3af3-8266-4e8f-97c4-e2681bad078b-tmp\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:37.951270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.951203 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-service-ca" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:19:37.951270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.951226 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-trusted-ca-bundle" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:19:37.951270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.951254 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-config" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:19:37.952884 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.952856 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-oauth-config" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:19:37.952884 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.952867 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3e3af3-8266-4e8f-97c4-e2681bad078b-kube-api-access-z5mkh" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "kube-api-access-z5mkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:19:37.953063 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:37.952894 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-serving-cert" pod "7f3e3af3-8266-4e8f-97c4-e2681bad078b" (UID: "7f3e3af3-8266-4e8f-97c4-e2681bad078b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:19:38.051409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.051373 6481 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-oauth-serving-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:38.051409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.051403 6481 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-serving-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:38.051409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.051415 6481 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-config\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:38.051634 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.051426 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mkh\" (UniqueName: \"kubernetes.io/projected/7f3e3af3-8266-4e8f-97c4-e2681bad078b-kube-api-access-z5mkh\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:38.051634 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.051437 6481 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f3e3af3-8266-4e8f-97c4-e2681bad078b-console-oauth-config\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:38.051634 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.051447 6481 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-trusted-ca-bundle\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:38.051634 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.051455 6481 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e3af3-8266-4e8f-97c4-e2681bad078b-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:19:38.443015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.442964 6481 generic.go:356] "Generic (PLEG): container finished" podID="7f3e3af3-8266-4e8f-97c4-e2681bad078b" containerID="8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114" exitCode=0 Oct 31 19:19:38.443196 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.443023 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7674df7bfb-55bm8" Oct 31 19:19:38.443196 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.443035 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7674df7bfb-55bm8" event={"ID":"7f3e3af3-8266-4e8f-97c4-e2681bad078b","Type":"ContainerDied","Data":"8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114"} Oct 31 19:19:38.443196 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.443085 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7674df7bfb-55bm8" event={"ID":"7f3e3af3-8266-4e8f-97c4-e2681bad078b","Type":"ContainerDied","Data":"daf11d9f375ef07a83e4e2f78c3394f37c14c9d7a11434089bfd2a05a51ef0e9"} Oct 31 19:19:38.443196 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.443108 6481 scope.go:122] "RemoveContainer" containerID="8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114" Oct 31 19:19:38.451146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.451120 6481 scope.go:122] "RemoveContainer" containerID="8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114" Oct 31 19:19:38.451448 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:19:38.451431 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114\": container with ID starting with 8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114 not found: ID does not exist" containerID="8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114" Oct 31 19:19:38.451503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.451457 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114"} err="failed to get container status \"8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114\": rpc error: code = NotFound desc = could not find container \"8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114\": container with ID starting with 8afbd38925a7a0b6dcfef15f4cbfe56072650dbec1a757dfe65872e6d3040114 not found: ID does not exist" Oct 31 19:19:38.595008 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.594958 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7674df7bfb-55bm8"] Oct 31 19:19:38.640376 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.640343 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7674df7bfb-55bm8"] Oct 31 19:19:38.876304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:19:38.876272 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3e3af3-8266-4e8f-97c4-e2681bad078b" path="/var/lib/kubelet/pods/7f3e3af3-8266-4e8f-97c4-e2681bad078b/volumes" Oct 31 19:19:57.323498 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:19:57.323463 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:20:27.322868 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:20:27.322785 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:20:57.322803 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:20:57.322769 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:21:14.449909 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.449876 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv"] Oct 31 19:21:14.450386 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.450187 6481 cpu_manager.go:414] "containerMap: removing container" podUID="7f3e3af3-8266-4e8f-97c4-e2681bad078b" containerName="console" Oct 31 19:21:14.450386 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.450197 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3e3af3-8266-4e8f-97c4-e2681bad078b" containerName="console" Oct 31 19:21:14.450386 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.450253 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3e3af3-8266-4e8f-97c4-e2681bad078b" containerName="console" Oct 31 19:21:14.453200 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.453185 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.463682 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.463657 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.463785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.463700 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.463785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.463760 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5lj\" (UniqueName: \"kubernetes.io/projected/fae97af9-a82c-4b50-bc6f-2694ce2860cf-kube-api-access-zg5lj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.547856 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.547825 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 19:21:14.548017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.547903 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 19:21:14.548017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.547901 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-mltsq\"" Oct 31 19:21:14.548602 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.548587 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv"] Oct 31 19:21:14.548989 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.548972 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w"] Oct 31 19:21:14.552181 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.552165 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.564580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.564554 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wdjt\" (UniqueName: \"kubernetes.io/projected/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-kube-api-access-4wdjt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.564692 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.564601 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.564692 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.564665 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.564803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.564698 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.564803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.564720 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.564803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.564737 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5lj\" (UniqueName: \"kubernetes.io/projected/fae97af9-a82c-4b50-bc6f-2694ce2860cf-kube-api-access-zg5lj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.565056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.565036 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.565101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.565081 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.602079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.602052 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-b29qw\"" Oct 31 19:21:14.621292 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.621255 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w"] Oct 31 19:21:14.650660 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.650632 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5lj\" (UniqueName: \"kubernetes.io/projected/fae97af9-a82c-4b50-bc6f-2694ce2860cf-kube-api-access-zg5lj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.665331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.665305 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.665485 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.665361 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wdjt\" (UniqueName: \"kubernetes.io/projected/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-kube-api-access-4wdjt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.665485 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.665413 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.665644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.665625 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.665731 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.665711 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.719675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.719641 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wdjt\" (UniqueName: \"kubernetes.io/projected/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-kube-api-access-4wdjt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.768729 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.768697 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:14.867394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.867363 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:14.909981 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.909954 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt"] Oct 31 19:21:14.916930 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.913354 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.963110 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.963083 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 19:21:14.967022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.966981 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.967081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.967063 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.967124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.967112 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4pdn\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-kube-api-access-r4pdn\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.967164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.967144 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f33277-a6a3-45d2-9371-49859f4b9420-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.967205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.967177 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.967240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.967228 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.967281 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.967252 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:14.981408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.981365 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 19:21:14.981408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.981367 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 19:21:14.981501 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.981452 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hnzhb\"" Oct 31 19:21:14.986030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.986017 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 19:21:14.986097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.986085 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 19:21:14.997241 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:14.997220 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt"] Oct 31 19:21:15.068528 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.068496 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f33277-a6a3-45d2-9371-49859f4b9420-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.068719 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.068542 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.068719 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.068569 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.068719 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.068595 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.068719 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.068669 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.068719 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.068704 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.068986 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.068743 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4pdn\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-kube-api-access-r4pdn\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.069806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.069745 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.071032 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.071007 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f33277-a6a3-45d2-9371-49859f4b9420-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.071195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.071176 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.071269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.071185 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.071269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.071224 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.082608 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.082592 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 19:21:15.122888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.122865 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.123656 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.123633 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4pdn\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-kube-api-access-r4pdn\") pod \"istiod-openshift-gateway-7876bfcb74-x4vnt\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.173114 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.173089 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv"] Oct 31 19:21:15.176181 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:21:15.176153 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae97af9_a82c_4b50_bc6f_2694ce2860cf.slice/crio-182c3612aa765a014cfa93713dcf1e93dac15dc26f74cd17fa193b0aefd8f20a WatchSource:0}: Error finding container 182c3612aa765a014cfa93713dcf1e93dac15dc26f74cd17fa193b0aefd8f20a: Status 404 returned error can't find the container with id 182c3612aa765a014cfa93713dcf1e93dac15dc26f74cd17fa193b0aefd8f20a Oct 31 19:21:15.225711 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.225683 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w"] Oct 31 19:21:15.228944 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:21:15.228921 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e23b6a_f76a_45e4_82dc_7f47812ff6b7.slice/crio-7b56427139d85ced79b62eab7134d1231c8d2e20228f1f3f1506ef5d281cffbb WatchSource:0}: Error finding container 7b56427139d85ced79b62eab7134d1231c8d2e20228f1f3f1506ef5d281cffbb: Status 404 returned error can't find the container with id 7b56427139d85ced79b62eab7134d1231c8d2e20228f1f3f1506ef5d281cffbb Oct 31 19:21:15.229043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.229011 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:15.591732 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.591656 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt"] Oct 31 19:21:15.594667 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:21:15.594642 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f33277_a6a3_45d2_9371_49859f4b9420.slice/crio-235d40ae2929de1f4df445d43de3c886118303db62a0ee08994098e5650a40d3 WatchSource:0}: Error finding container 235d40ae2929de1f4df445d43de3c886118303db62a0ee08994098e5650a40d3: Status 404 returned error can't find the container with id 235d40ae2929de1f4df445d43de3c886118303db62a0ee08994098e5650a40d3 Oct 31 19:21:15.763691 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.763653 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" event={"ID":"fae97af9-a82c-4b50-bc6f-2694ce2860cf","Type":"ContainerStarted","Data":"182c3612aa765a014cfa93713dcf1e93dac15dc26f74cd17fa193b0aefd8f20a"} Oct 31 19:21:15.764758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.764733 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" event={"ID":"e5f33277-a6a3-45d2-9371-49859f4b9420","Type":"ContainerStarted","Data":"235d40ae2929de1f4df445d43de3c886118303db62a0ee08994098e5650a40d3"} Oct 31 19:21:15.765732 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:15.765708 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" event={"ID":"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7","Type":"ContainerStarted","Data":"7b56427139d85ced79b62eab7134d1231c8d2e20228f1f3f1506ef5d281cffbb"} Oct 31 19:21:20.402710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:20.402668 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:21:20.403132 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:20.402761 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:21:20.789861 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:20.789820 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" event={"ID":"e5f33277-a6a3-45d2-9371-49859f4b9420","Type":"ContainerStarted","Data":"7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915"} Oct 31 19:21:20.790054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:20.789932 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:21.797216 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:21.797180 6481 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-x4vnt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:21:21.797650 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:21.797237 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" podUID="e5f33277-a6a3-45d2-9371-49859f4b9420" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:21:22.804657 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:22.804564 6481 generic.go:356] "Generic (PLEG): container finished" podID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerID="950dd8f42967653be02252da06431ea922f5512d4639b8526397928fc0f28195" exitCode=0 Oct 31 19:21:22.804657 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:22.804624 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" event={"ID":"fae97af9-a82c-4b50-bc6f-2694ce2860cf","Type":"ContainerDied","Data":"950dd8f42967653be02252da06431ea922f5512d4639b8526397928fc0f28195"} Oct 31 19:21:22.806207 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:22.806188 6481 generic.go:356] "Generic (PLEG): container finished" podID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerID="d6a54dfd9d8db7830c40e9194f17e53c732996774cf88d45823085cf89bd5c99" exitCode=0 Oct 31 19:21:22.806281 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:22.806222 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" event={"ID":"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7","Type":"ContainerDied","Data":"d6a54dfd9d8db7830c40e9194f17e53c732996774cf88d45823085cf89bd5c99"} Oct 31 19:21:22.924774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:22.924723 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" podStartSLOduration=4.119661855 podStartE2EDuration="8.92470749s" totalImagesPullingTime="4.805045635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:20.898614747 +0000 UTC m=+252.138837879" watchObservedRunningTime="2026-10-31 19:21:22.92470749 +0000 UTC m=+254.164930619" Oct 31 19:21:24.796763 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:24.796705 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:21:26.823846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:26.823814 6481 generic.go:356] "Generic (PLEG): container finished" podID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerID="9c2e0cfe7c1320e2af02743ad1fd4bf98c946cbbce95a466b0ea32293b516196" exitCode=0 Oct 31 19:21:26.824306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:26.823901 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" event={"ID":"fae97af9-a82c-4b50-bc6f-2694ce2860cf","Type":"ContainerDied","Data":"9c2e0cfe7c1320e2af02743ad1fd4bf98c946cbbce95a466b0ea32293b516196"} Oct 31 19:21:26.825876 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:26.825783 6481 generic.go:356] "Generic (PLEG): container finished" podID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerID="e520efdbd6934f3a743b4bc50181e882ca0b8bf1fc416ec305f817a5002bcab9" exitCode=0 Oct 31 19:21:26.825876 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:26.825818 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" event={"ID":"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7","Type":"ContainerDied","Data":"e520efdbd6934f3a743b4bc50181e882ca0b8bf1fc416ec305f817a5002bcab9"} Oct 31 19:21:27.323822 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:21:27.323790 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:21:34.858852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:34.858813 6481 generic.go:356] "Generic (PLEG): container finished" podID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerID="2b270586d61c73f174857768cf273d08411311f836a8363f7efd708fdb2ea1dd" exitCode=0 Oct 31 19:21:34.859303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:34.858892 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" event={"ID":"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7","Type":"ContainerDied","Data":"2b270586d61c73f174857768cf273d08411311f836a8363f7efd708fdb2ea1dd"} Oct 31 19:21:34.860853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:34.860827 6481 generic.go:356] "Generic (PLEG): container finished" podID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerID="41f249d198eeffd996b60db7defe577a20665bf2a5e4dcea3bb516abf2ede3aa" exitCode=0 Oct 31 19:21:34.860965 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:34.860894 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" event={"ID":"fae97af9-a82c-4b50-bc6f-2694ce2860cf","Type":"ContainerDied","Data":"41f249d198eeffd996b60db7defe577a20665bf2a5e4dcea3bb516abf2ede3aa"} Oct 31 19:21:36.001551 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.001526 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:36.025024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.024983 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:36.059456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.059432 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-bundle\") pod \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " Oct 31 19:21:36.059613 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.059470 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae97af9-a82c-4b50-bc6f-2694ce2860cf-kube-api-access-zg5lj\" (UniqueName: \"kubernetes.io/projected/fae97af9-a82c-4b50-bc6f-2694ce2860cf-kube-api-access-zg5lj\") pod \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " Oct 31 19:21:36.059613 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.059531 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-util\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-util\") pod \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\" (UID: \"fae97af9-a82c-4b50-bc6f-2694ce2860cf\") " Oct 31 19:21:36.059936 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.059899 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-bundle" pod "fae97af9-a82c-4b50-bc6f-2694ce2860cf" (UID: "fae97af9-a82c-4b50-bc6f-2694ce2860cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:36.061736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.061715 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae97af9-a82c-4b50-bc6f-2694ce2860cf-kube-api-access-zg5lj" pod "fae97af9-a82c-4b50-bc6f-2694ce2860cf" (UID: "fae97af9-a82c-4b50-bc6f-2694ce2860cf"). InnerVolumeSpecName "kube-api-access-zg5lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:36.064791 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.064763 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-util" pod "fae97af9-a82c-4b50-bc6f-2694ce2860cf" (UID: "fae97af9-a82c-4b50-bc6f-2694ce2860cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:36.160483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.160411 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-util\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-util\") pod \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " Oct 31 19:21:36.160483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.160446 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-bundle\") pod \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " Oct 31 19:21:36.160483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.160483 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-kube-api-access-4wdjt\" (UniqueName: \"kubernetes.io/projected/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-kube-api-access-4wdjt\") pod \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\" (UID: \"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7\") " Oct 31 19:21:36.160762 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.160619 6481 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-util\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:36.160762 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.160632 6481 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fae97af9-a82c-4b50-bc6f-2694ce2860cf-bundle\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:36.160762 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.160641 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5lj\" (UniqueName: \"kubernetes.io/projected/fae97af9-a82c-4b50-bc6f-2694ce2860cf-kube-api-access-zg5lj\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:36.161176 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.161151 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-bundle" pod "b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" (UID: "b0e23b6a-f76a-45e4-82dc-7f47812ff6b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:36.162464 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.162444 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-kube-api-access-4wdjt" pod "b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" (UID: "b0e23b6a-f76a-45e4-82dc-7f47812ff6b7"). InnerVolumeSpecName "kube-api-access-4wdjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:21:36.164605 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.164582 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-util" pod "b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" (UID: "b0e23b6a-f76a-45e4-82dc-7f47812ff6b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:21:36.261612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.261572 6481 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-util\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:36.261612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.261605 6481 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-bundle\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:36.261612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.261618 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wdjt\" (UniqueName: \"kubernetes.io/projected/b0e23b6a-f76a-45e4-82dc-7f47812ff6b7-kube-api-access-4wdjt\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:21:36.869959 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.869932 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" Oct 31 19:21:36.870147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.869956 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fskxv" event={"ID":"fae97af9-a82c-4b50-bc6f-2694ce2860cf","Type":"ContainerDied","Data":"182c3612aa765a014cfa93713dcf1e93dac15dc26f74cd17fa193b0aefd8f20a"} Oct 31 19:21:36.870147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.870017 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="182c3612aa765a014cfa93713dcf1e93dac15dc26f74cd17fa193b0aefd8f20a" Oct 31 19:21:36.871924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.871902 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" Oct 31 19:21:36.878527 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.878499 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdh66w" event={"ID":"b0e23b6a-f76a-45e4-82dc-7f47812ff6b7","Type":"ContainerDied","Data":"7b56427139d85ced79b62eab7134d1231c8d2e20228f1f3f1506ef5d281cffbb"} Oct 31 19:21:36.878665 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:36.878548 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b56427139d85ced79b62eab7134d1231c8d2e20228f1f3f1506ef5d281cffbb" Oct 31 19:21:55.422162 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.421952 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm"] Oct 31 19:21:55.422795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422771 6481 cpu_manager.go:414] "containerMap: removing container" podUID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerName="util" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422795 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerName="util" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422810 6481 cpu_manager.go:414] "containerMap: removing container" podUID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerName="util" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422818 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerName="util" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422851 6481 cpu_manager.go:414] "containerMap: removing container" podUID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerName="pull" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422859 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerName="pull" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422873 6481 cpu_manager.go:414] "containerMap: removing container" podUID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerName="pull" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422879 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerName="pull" Oct 31 19:21:55.422892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422892 6481 cpu_manager.go:414] "containerMap: removing container" podUID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerName="extract" Oct 31 19:21:55.423291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422899 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerName="extract" Oct 31 19:21:55.423291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422911 6481 cpu_manager.go:414] "containerMap: removing container" podUID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerName="extract" Oct 31 19:21:55.423291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.422918 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerName="extract" Oct 31 19:21:55.423291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.423017 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae97af9-a82c-4b50-bc6f-2694ce2860cf" containerName="extract" Oct 31 19:21:55.423291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.423032 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e23b6a-f76a-45e4-82dc-7f47812ff6b7" containerName="extract" Oct 31 19:21:55.426176 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.426155 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.475009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.474974 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 19:21:55.475125 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.475044 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 19:21:55.475264 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.475247 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 19:21:55.475542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.475526 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7fffj\"" Oct 31 19:21:55.475542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.475527 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 19:21:55.521560 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.521536 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 19:21:55.528341 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.528317 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.528437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.528353 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.528437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.528386 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.528536 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.528517 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.528595 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.528546 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.528668 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.528646 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2sl\" (UniqueName: \"kubernetes.io/projected/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-kube-api-access-8g2sl\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.543442 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.543422 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm"] Oct 31 19:21:55.630042 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.629989 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.630222 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.630060 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.630222 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.630085 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.630222 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.630123 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2sl\" (UniqueName: \"kubernetes.io/projected/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-kube-api-access-8g2sl\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.630222 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.630166 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.630222 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.630200 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.630564 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.630536 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-manifests\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.632418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.632393 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-tmp\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.632569 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.632553 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.632631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.632608 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.632686 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.632633 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-metrics-cert\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.686842 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.686789 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2sl\" (UniqueName: \"kubernetes.io/projected/a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9-kube-api-access-8g2sl\") pod \"opendatahub-operator-controller-manager-dc8db5648-zgwhm\" (UID: \"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:55.739751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:55.739717 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:21:56.231419 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:56.231382 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm"] Oct 31 19:21:56.235299 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:21:56.235269 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda873d3e3_fae4_4065_8e3a_40e2a6c3a6b9.slice/crio-984acba3f338686c275c4c6e00837a91cffee02a0dd170e1487b5686b75492af WatchSource:0}: Error finding container 984acba3f338686c275c4c6e00837a91cffee02a0dd170e1487b5686b75492af: Status 404 returned error can't find the container with id 984acba3f338686c275c4c6e00837a91cffee02a0dd170e1487b5686b75492af Oct 31 19:21:56.945047 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:56.945011 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" event={"ID":"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9","Type":"ContainerStarted","Data":"984acba3f338686c275c4c6e00837a91cffee02a0dd170e1487b5686b75492af"} Oct 31 19:21:57.325595 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:21:57.325560 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:21:59.958067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:21:59.958031 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" event={"ID":"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9","Type":"ContainerStarted","Data":"9c6759043485efc7a5d0f3608e2eb01023a59a8f76b22feea0884d4346692fc9"} Oct 31 19:22:00.963066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:00.963032 6481 generic.go:356] "Generic (PLEG): container finished" podID="a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9" containerID="9c6759043485efc7a5d0f3608e2eb01023a59a8f76b22feea0884d4346692fc9" exitCode=0 Oct 31 19:22:00.963518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:00.963112 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" event={"ID":"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9","Type":"ContainerDied","Data":"9c6759043485efc7a5d0f3608e2eb01023a59a8f76b22feea0884d4346692fc9"} Oct 31 19:22:01.968851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:01.968822 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" event={"ID":"a873d3e3-fae4-4065-8e3a-40e2a6c3a6b9","Type":"ContainerStarted","Data":"325f9d8debb08d1cc0469be61426fd7e4b7ad831e60c5342ab12d1ef9c1b125e"} Oct 31 19:22:01.969191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:01.968933 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:22:02.195645 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:02.195536 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" podStartSLOduration=2.3804566019999998 podStartE2EDuration="7.195518363s" totalImagesPullingTime="4.408587762s" totalInitContainerRuntime="406.473999ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:02.149049052 +0000 UTC m=+293.389272182" watchObservedRunningTime="2026-10-31 19:22:02.195518363 +0000 UTC m=+293.435741493" Oct 31 19:22:12.975400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:12.975371 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dc8db5648-zgwhm" Oct 31 19:22:25.559058 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.559028 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w"] Oct 31 19:22:25.561303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.561287 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.610319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.610292 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 19:22:25.610319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.610292 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 19:22:25.656562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.656530 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 19:22:25.656703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.656670 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8rcsr\"" Oct 31 19:22:25.656743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.656730 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 19:22:25.657544 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.657524 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w"] Oct 31 19:22:25.677419 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.677390 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93b628d-f933-44de-a0b9-42301b2524cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.677548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.677424 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dch4s\" (UniqueName: \"kubernetes.io/projected/b93b628d-f933-44de-a0b9-42301b2524cc-kube-api-access-dch4s\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.677548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.677446 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b93b628d-f933-44de-a0b9-42301b2524cc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.778389 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.778352 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93b628d-f933-44de-a0b9-42301b2524cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.778389 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.778395 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dch4s\" (UniqueName: \"kubernetes.io/projected/b93b628d-f933-44de-a0b9-42301b2524cc-kube-api-access-dch4s\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.778624 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.778417 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b93b628d-f933-44de-a0b9-42301b2524cc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.779066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.779048 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b93b628d-f933-44de-a0b9-42301b2524cc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.780764 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.780746 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b93b628d-f933-44de-a0b9-42301b2524cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.831980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.831909 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dch4s\" (UniqueName: \"kubernetes.io/projected/b93b628d-f933-44de-a0b9-42301b2524cc-kube-api-access-dch4s\") pod \"kuadrant-console-plugin-66c4fb575d-pw67w\" (UID: \"b93b628d-f933-44de-a0b9-42301b2524cc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:25.876448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:25.876421 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" Oct 31 19:22:26.234014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:26.233972 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w"] Oct 31 19:22:26.237191 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:22:26.237157 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93b628d_f933_44de_a0b9_42301b2524cc.slice/crio-16a0290b0c8f6ab9998dca404961f5e40e031cb1cca01f054b9d86d21c494816 WatchSource:0}: Error finding container 16a0290b0c8f6ab9998dca404961f5e40e031cb1cca01f054b9d86d21c494816: Status 404 returned error can't find the container with id 16a0290b0c8f6ab9998dca404961f5e40e031cb1cca01f054b9d86d21c494816 Oct 31 19:22:27.060303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:27.060267 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" event={"ID":"b93b628d-f933-44de-a0b9-42301b2524cc","Type":"ContainerStarted","Data":"16a0290b0c8f6ab9998dca404961f5e40e031cb1cca01f054b9d86d21c494816"} Oct 31 19:22:27.324217 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:22:27.324133 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:22:30.015274 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.015235 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz"] Oct 31 19:22:30.026291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.026262 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.115872 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.115829 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea81bab8-4e72-4032-83bd-b559eab23229-tls-certs\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.116078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.115882 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea81bab8-4e72-4032-83bd-b559eab23229-tmp\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.116078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.116038 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58fp\" (UniqueName: \"kubernetes.io/projected/ea81bab8-4e72-4032-83bd-b559eab23229-kube-api-access-n58fp\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.121766 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.121739 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz"] Oct 31 19:22:30.154441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.154400 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 19:22:30.167609 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.167581 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7q2np\"" Oct 31 19:22:30.167609 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.167589 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 19:22:30.217063 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.217016 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58fp\" (UniqueName: \"kubernetes.io/projected/ea81bab8-4e72-4032-83bd-b559eab23229-kube-api-access-n58fp\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.217238 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.217135 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea81bab8-4e72-4032-83bd-b559eab23229-tls-certs\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.217238 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.217184 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea81bab8-4e72-4032-83bd-b559eab23229-tmp\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.220045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.220016 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea81bab8-4e72-4032-83bd-b559eab23229-tmp\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.220258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.220239 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea81bab8-4e72-4032-83bd-b559eab23229-tls-certs\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.271111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.271040 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58fp\" (UniqueName: \"kubernetes.io/projected/ea81bab8-4e72-4032-83bd-b559eab23229-kube-api-access-n58fp\") pod \"kube-auth-proxy-9b655b5cf-ndjvz\" (UID: \"ea81bab8-4e72-4032-83bd-b559eab23229\") " pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.340228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.340195 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" Oct 31 19:22:30.721248 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:30.721213 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz"] Oct 31 19:22:30.724789 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:22:30.724752 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea81bab8_4e72_4032_83bd_b559eab23229.slice/crio-a9cc73c3547bbc3d9aa27faa69fd9dfce0f89b74ade2ea56f756fbf56ab495d6 WatchSource:0}: Error finding container a9cc73c3547bbc3d9aa27faa69fd9dfce0f89b74ade2ea56f756fbf56ab495d6: Status 404 returned error can't find the container with id a9cc73c3547bbc3d9aa27faa69fd9dfce0f89b74ade2ea56f756fbf56ab495d6 Oct 31 19:22:31.083004 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:31.082948 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" event={"ID":"ea81bab8-4e72-4032-83bd-b559eab23229","Type":"ContainerStarted","Data":"a9cc73c3547bbc3d9aa27faa69fd9dfce0f89b74ade2ea56f756fbf56ab495d6"} Oct 31 19:22:32.091885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:32.091799 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" event={"ID":"b93b628d-f933-44de-a0b9-42301b2524cc","Type":"ContainerStarted","Data":"8c2e9d463847f0db89b50054a51b9cb49300cf6e41e49d2f623420dfd0fb7ab9"} Oct 31 19:22:32.203538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:32.203481 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pw67w" podStartSLOduration=1.644242352 podStartE2EDuration="7.203465959s" totalImagesPullingTime="5.559223607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:32.202933409 +0000 UTC m=+323.443156539" watchObservedRunningTime="2026-10-31 19:22:32.203465959 +0000 UTC m=+323.443689082" Oct 31 19:22:34.106743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:34.106697 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" event={"ID":"ea81bab8-4e72-4032-83bd-b559eab23229","Type":"ContainerStarted","Data":"6d669265bd91ce13bb16a837ad67cfff6cf45b06779e05ca0a18d7f0a0b285d8"} Oct 31 19:22:34.230368 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:22:34.230304 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9b655b5cf-ndjvz" podStartSLOduration=2.445756134 podStartE2EDuration="5.23028515s" totalImagesPullingTime="2.784529016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:22:34.229086144 +0000 UTC m=+325.469309273" watchObservedRunningTime="2026-10-31 19:22:34.23028515 +0000 UTC m=+325.470508278" Oct 31 19:22:57.323963 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:22:57.323929 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:23:27.324500 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:23:27.324413 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:23:55.835504 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:55.835471 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5"] Oct 31 19:23:55.838639 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:55.838621 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:55.935027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:55.934958 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5"] Oct 31 19:23:56.010849 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.010813 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.010849 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.010857 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.011081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.010976 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.011081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.011047 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.011081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.011073 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.011173 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.011103 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.011208 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.011182 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkt9\" (UniqueName: \"kubernetes.io/projected/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-kube-api-access-cvkt9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.111811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.111733 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkt9\" (UniqueName: \"kubernetes.io/projected/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-kube-api-access-cvkt9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.111811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.111772 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.111811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.111802 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.112059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.111866 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.112059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.111902 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.112059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.111924 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.112059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.111965 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.112661 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.112634 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.114319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.114245 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.114319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.114267 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.114500 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.114474 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.114645 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.114621 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.169156 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.169124 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkt9\" (UniqueName: \"kubernetes.io/projected/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-kube-api-access-cvkt9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.169481 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.169463 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31be8330-7a6e-46fa-aa22-da8cf0c90c3c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vjzx5\" (UID: \"31be8330-7a6e-46fa-aa22-da8cf0c90c3c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.449748 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.449670 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:56.858744 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.858702 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5"] Oct 31 19:23:56.861732 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:23:56.861708 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31be8330_7a6e_46fa_aa22_da8cf0c90c3c.slice/crio-f3b7f94273e20d8b1d3dbcf9a7aa7b111951455f2bdcc88307c559465b85dc5b WatchSource:0}: Error finding container f3b7f94273e20d8b1d3dbcf9a7aa7b111951455f2bdcc88307c559465b85dc5b: Status 404 returned error can't find the container with id f3b7f94273e20d8b1d3dbcf9a7aa7b111951455f2bdcc88307c559465b85dc5b Oct 31 19:23:56.863877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.863842 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:23:56.863970 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:56.863919 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:23:57.323560 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:23:57.323517 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:23:57.427348 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:57.427306 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" event={"ID":"31be8330-7a6e-46fa-aa22-da8cf0c90c3c","Type":"ContainerStarted","Data":"f69d04deef3cc5a50fc95f6f17d35242265e7cd5b541d3d8bd4a16aeb3ab962e"} Oct 31 19:23:57.427574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:57.427358 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" event={"ID":"31be8330-7a6e-46fa-aa22-da8cf0c90c3c","Type":"ContainerStarted","Data":"f3b7f94273e20d8b1d3dbcf9a7aa7b111951455f2bdcc88307c559465b85dc5b"} Oct 31 19:23:57.427574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:57.427490 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:57.429180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:57.429152 6481 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vjzx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:23:57.429284 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:57.429211 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" podUID="31be8330-7a6e-46fa-aa22-da8cf0c90c3c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:23:57.569124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:57.569061 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" podStartSLOduration=2.569047196 podStartE2EDuration="2.569047196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:57.536799221 +0000 UTC m=+408.777022350" watchObservedRunningTime="2026-10-31 19:23:57.569047196 +0000 UTC m=+408.809270327" Oct 31 19:23:58.432347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:58.432319 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vjzx5" Oct 31 19:23:58.635638 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:58.635607 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt"] Oct 31 19:23:58.635821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:58.635800 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" podUID="e5f33277-a6a3-45d2-9371-49859f4b9420" containerName="discovery" containerID="cri-o://7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915" gracePeriod=30 Oct 31 19:23:58.879514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:58.879489 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:23:59.036258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036227 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-kubeconfig\") pod \"e5f33277-a6a3-45d2-9371-49859f4b9420\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " Oct 31 19:23:59.036425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036268 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-dns-cert\") pod \"e5f33277-a6a3-45d2-9371-49859f4b9420\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " Oct 31 19:23:59.036425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036302 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-kube-api-access-r4pdn\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-kube-api-access-r4pdn\") pod \"e5f33277-a6a3-45d2-9371-49859f4b9420\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " Oct 31 19:23:59.036425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036354 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f33277-a6a3-45d2-9371-49859f4b9420-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f33277-a6a3-45d2-9371-49859f4b9420-local-certs\") pod \"e5f33277-a6a3-45d2-9371-49859f4b9420\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " Oct 31 19:23:59.036425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036388 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-ca-configmap\") pod \"e5f33277-a6a3-45d2-9371-49859f4b9420\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " Oct 31 19:23:59.036425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036422 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-token\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-token\") pod \"e5f33277-a6a3-45d2-9371-49859f4b9420\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " Oct 31 19:23:59.036678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036444 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-cacerts\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-cacerts\") pod \"e5f33277-a6a3-45d2-9371-49859f4b9420\" (UID: \"e5f33277-a6a3-45d2-9371-49859f4b9420\") " Oct 31 19:23:59.036889 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.036843 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-ca-configmap" pod "e5f33277-a6a3-45d2-9371-49859f4b9420" (UID: "e5f33277-a6a3-45d2-9371-49859f4b9420"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:23:59.038780 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.038749 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f33277-a6a3-45d2-9371-49859f4b9420-local-certs" pod "e5f33277-a6a3-45d2-9371-49859f4b9420" (UID: "e5f33277-a6a3-45d2-9371-49859f4b9420"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:59.038780 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.038758 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-dns-cert" pod "e5f33277-a6a3-45d2-9371-49859f4b9420" (UID: "e5f33277-a6a3-45d2-9371-49859f4b9420"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:23:59.038953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.038825 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-kubeconfig" pod "e5f33277-a6a3-45d2-9371-49859f4b9420" (UID: "e5f33277-a6a3-45d2-9371-49859f4b9420"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:23:59.038953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.038945 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-kube-api-access-r4pdn" pod "e5f33277-a6a3-45d2-9371-49859f4b9420" (UID: "e5f33277-a6a3-45d2-9371-49859f4b9420"). InnerVolumeSpecName "kube-api-access-r4pdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:59.039081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.039042 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-cacerts" pod "e5f33277-a6a3-45d2-9371-49859f4b9420" (UID: "e5f33277-a6a3-45d2-9371-49859f4b9420"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:23:59.039140 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.039108 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-token" pod "e5f33277-a6a3-45d2-9371-49859f4b9420" (UID: "e5f33277-a6a3-45d2-9371-49859f4b9420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:59.137672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.137636 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:59.137672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.137676 6481 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-cacerts\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:59.137874 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.137693 6481 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-kubeconfig\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:59.137874 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.137711 6481 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-dns-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:59.137874 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.137727 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4pdn\" (UniqueName: \"kubernetes.io/projected/e5f33277-a6a3-45d2-9371-49859f4b9420-kube-api-access-r4pdn\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:59.137874 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.137744 6481 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f33277-a6a3-45d2-9371-49859f4b9420-local-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:59.137874 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.137758 6481 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5f33277-a6a3-45d2-9371-49859f4b9420-istio-csr-ca-configmap\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:59.436702 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.436607 6481 generic.go:356] "Generic (PLEG): container finished" podID="e5f33277-a6a3-45d2-9371-49859f4b9420" containerID="7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915" exitCode=0 Oct 31 19:23:59.436702 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.436659 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" Oct 31 19:23:59.437228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.436696 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" event={"ID":"e5f33277-a6a3-45d2-9371-49859f4b9420","Type":"ContainerDied","Data":"7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915"} Oct 31 19:23:59.437228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.436754 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt" event={"ID":"e5f33277-a6a3-45d2-9371-49859f4b9420","Type":"ContainerDied","Data":"235d40ae2929de1f4df445d43de3c886118303db62a0ee08994098e5650a40d3"} Oct 31 19:23:59.437228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.436778 6481 scope.go:122] "RemoveContainer" containerID="7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915" Oct 31 19:23:59.445097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.445080 6481 scope.go:122] "RemoveContainer" containerID="7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915" Oct 31 19:23:59.445369 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:23:59.445350 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915\": container with ID starting with 7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915 not found: ID does not exist" containerID="7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915" Oct 31 19:23:59.445446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.445381 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915"} err="failed to get container status \"7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915\": rpc error: code = NotFound desc = could not find container \"7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915\": container with ID starting with 7282c3131feaa1e24c4a4ba67dc9f4bc9080c818c35ade41859e9aedefdde915 not found: ID does not exist" Oct 31 19:23:59.590442 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.590402 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt"] Oct 31 19:23:59.641135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:23:59.637035 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x4vnt"] Oct 31 19:24:00.878045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:00.877988 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f33277-a6a3-45d2-9371-49859f4b9420" path="/var/lib/kubelet/pods/e5f33277-a6a3-45d2-9371-49859f4b9420/volumes" Oct 31 19:24:01.940725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.940692 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b5mbb"] Oct 31 19:24:01.941120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.941007 6481 cpu_manager.go:414] "containerMap: removing container" podUID="e5f33277-a6a3-45d2-9371-49859f4b9420" containerName="discovery" Oct 31 19:24:01.941120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.941018 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f33277-a6a3-45d2-9371-49859f4b9420" containerName="discovery" Oct 31 19:24:01.941120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.941082 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f33277-a6a3-45d2-9371-49859f4b9420" containerName="discovery" Oct 31 19:24:01.944981 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.944956 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:01.961911 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.961887 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2-cert\") pod \"kserve-controller-manager-776cc66d47-b5mbb\" (UID: \"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:01.962020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.961926 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzvh\" (UniqueName: \"kubernetes.io/projected/6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2-kube-api-access-9tzvh\") pod \"kserve-controller-manager-776cc66d47-b5mbb\" (UID: \"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:01.996477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:01.996443 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b5mbb"] Oct 31 19:24:02.041472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.041437 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 19:24:02.041472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.041472 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 19:24:02.041697 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.041475 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 19:24:02.041697 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.041568 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mwzlh\"" Oct 31 19:24:02.062317 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.062291 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2-cert\") pod \"kserve-controller-manager-776cc66d47-b5mbb\" (UID: \"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:02.062447 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.062330 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzvh\" (UniqueName: \"kubernetes.io/projected/6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2-kube-api-access-9tzvh\") pod \"kserve-controller-manager-776cc66d47-b5mbb\" (UID: \"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:02.064755 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.064736 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2-cert\") pod \"kserve-controller-manager-776cc66d47-b5mbb\" (UID: \"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:02.115272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.115245 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzvh\" (UniqueName: \"kubernetes.io/projected/6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2-kube-api-access-9tzvh\") pod \"kserve-controller-manager-776cc66d47-b5mbb\" (UID: \"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:02.162922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.162884 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mtl9j"] Oct 31 19:24:02.166200 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.166184 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.218358 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.218318 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 19:24:02.218358 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.218331 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-d5kz8\"" Oct 31 19:24:02.219220 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.219197 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mtl9j"] Oct 31 19:24:02.257491 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.257464 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:02.264316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.264291 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f-tls-certs\") pod \"model-serving-api-75bd89fdfc-mtl9j\" (UID: \"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f\") " pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.264423 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.264353 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq69h\" (UniqueName: \"kubernetes.io/projected/ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f-kube-api-access-lq69h\") pod \"model-serving-api-75bd89fdfc-mtl9j\" (UID: \"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f\") " pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.365446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.365417 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq69h\" (UniqueName: \"kubernetes.io/projected/ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f-kube-api-access-lq69h\") pod \"model-serving-api-75bd89fdfc-mtl9j\" (UID: \"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f\") " pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.365638 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.365478 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f-tls-certs\") pod \"model-serving-api-75bd89fdfc-mtl9j\" (UID: \"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f\") " pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.367951 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.367929 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f-tls-certs\") pod \"model-serving-api-75bd89fdfc-mtl9j\" (UID: \"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f\") " pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.418563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.418531 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq69h\" (UniqueName: \"kubernetes.io/projected/ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f-kube-api-access-lq69h\") pod \"model-serving-api-75bd89fdfc-mtl9j\" (UID: \"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f\") " pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.478303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.478227 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:02.617883 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.617849 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b5mbb"] Oct 31 19:24:02.620779 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:24:02.620754 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea7565c_19d2_4bd9_b37a_9cb2b15dd4c2.slice/crio-a0815d467113139e6ab49f86424fd3232be4cca8d7832db15cc29c94ada16300 WatchSource:0}: Error finding container a0815d467113139e6ab49f86424fd3232be4cca8d7832db15cc29c94ada16300: Status 404 returned error can't find the container with id a0815d467113139e6ab49f86424fd3232be4cca8d7832db15cc29c94ada16300 Oct 31 19:24:02.621931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.621915 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:24:02.839297 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:02.839226 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mtl9j"] Oct 31 19:24:02.842201 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:24:02.842177 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba68fe55_84a9_4ae2_b46e_9eb0bc4ab71f.slice/crio-209064fce31c76aba0429de9eb9aa7c7a2eab32f5296be7e09b36821d377208a WatchSource:0}: Error finding container 209064fce31c76aba0429de9eb9aa7c7a2eab32f5296be7e09b36821d377208a: Status 404 returned error can't find the container with id 209064fce31c76aba0429de9eb9aa7c7a2eab32f5296be7e09b36821d377208a Oct 31 19:24:03.455319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:03.455284 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" event={"ID":"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2","Type":"ContainerStarted","Data":"a0815d467113139e6ab49f86424fd3232be4cca8d7832db15cc29c94ada16300"} Oct 31 19:24:03.456349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:03.456324 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" event={"ID":"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f","Type":"ContainerStarted","Data":"209064fce31c76aba0429de9eb9aa7c7a2eab32f5296be7e09b36821d377208a"} Oct 31 19:24:05.468724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:05.468675 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" event={"ID":"ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f","Type":"ContainerStarted","Data":"bae9c4d955ba6a37c2eb6e77df3a207ba55101284c196e72ed7fd2f59c097e20"} Oct 31 19:24:05.469140 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:05.468777 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:05.470601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:05.470576 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" event={"ID":"6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2","Type":"ContainerStarted","Data":"7a99a6bbeba92a33d3a6f2a7859327b4a7822ed732a905ea3512ef9aaa069ca6"} Oct 31 19:24:05.470753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:05.470649 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:05.575553 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:05.575497 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" podStartSLOduration=1.083545738 podStartE2EDuration="3.575480012s" totalImagesPullingTime="2.491934274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:05.574429202 +0000 UTC m=+416.814652328" watchObservedRunningTime="2026-10-31 19:24:05.575480012 +0000 UTC m=+416.815703137" Oct 31 19:24:05.681813 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:05.681699 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" podStartSLOduration=1.929266457 podStartE2EDuration="4.68168322s" totalImagesPullingTime="2.752416763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:05.680638881 +0000 UTC m=+416.920862008" watchObservedRunningTime="2026-10-31 19:24:05.68168322 +0000 UTC m=+416.921906349" Oct 31 19:24:16.478657 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:16.478625 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-mtl9j" Oct 31 19:24:27.324125 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:24:27.324090 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:24:36.480864 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:24:36.480833 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b5mbb" Oct 31 19:24:57.323538 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:24:57.323462 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:25:27.324074 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:25:27.324030 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:25:57.323701 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:25:57.323604 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:26:27.324118 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:26:27.324038 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:26:57.323560 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:26:57.323521 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:27:27.323482 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:27:27.323443 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:27:57.323563 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:27:57.323488 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:28:27.324765 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:28:27.324728 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:28:57.324532 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:28:57.324495 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:29:27.322877 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:29:27.322796 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:29:57.324366 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:29:57.324319 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:30:27.323960 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:30:27.323923 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:30:57.324074 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:30:57.323946 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:31:27.323939 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:31:27.323899 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:31:57.323281 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:31:57.323245 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:32:27.322883 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:32:27.322792 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:32:57.324457 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:32:57.324419 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:33:10.748713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.748682 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z27b4"] Oct 31 19:33:10.751801 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.751785 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:10.799958 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.799929 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 19:33:10.804336 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.804310 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z27b4"] Oct 31 19:33:10.898971 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.898936 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwks\" (UniqueName: \"kubernetes.io/projected/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-kube-api-access-mcwks\") pod \"limitador-limitador-74dfcfb894-z27b4\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:10.899153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.899024 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-config-file\") pod \"limitador-limitador-74dfcfb894-z27b4\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:10.991513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.991482 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z27b4"] Oct 31 19:33:10.999843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.999799 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-config-file\") pod \"limitador-limitador-74dfcfb894-z27b4\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:10.999917 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:10.999864 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwks\" (UniqueName: \"kubernetes.io/projected/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-kube-api-access-mcwks\") pod \"limitador-limitador-74dfcfb894-z27b4\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:11.000487 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:11.000464 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-config-file\") pod \"limitador-limitador-74dfcfb894-z27b4\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:11.053012 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:11.052977 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwks\" (UniqueName: \"kubernetes.io/projected/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-kube-api-access-mcwks\") pod \"limitador-limitador-74dfcfb894-z27b4\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:11.063802 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:11.063778 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:11.423629 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:11.423555 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z27b4"] Oct 31 19:33:11.426699 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:33:11.426673 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aa697b_d3ff_4b57_b49e_72e5cf32a9ca.slice/crio-78c5ed05eff41b91c0446e17e23213aba6c3a5572833d409e2ccb81db54805d1 WatchSource:0}: Error finding container 78c5ed05eff41b91c0446e17e23213aba6c3a5572833d409e2ccb81db54805d1: Status 404 returned error can't find the container with id 78c5ed05eff41b91c0446e17e23213aba6c3a5572833d409e2ccb81db54805d1 Oct 31 19:33:11.428662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:11.428644 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:33:11.623776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:11.623738 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" event={"ID":"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca","Type":"ContainerStarted","Data":"78c5ed05eff41b91c0446e17e23213aba6c3a5572833d409e2ccb81db54805d1"} Oct 31 19:33:14.898473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:14.898441 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bkkcb"] Oct 31 19:33:14.902124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:14.902098 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" Oct 31 19:33:14.955074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:14.954411 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bkkcb"] Oct 31 19:33:14.999160 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:14.999125 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ldbjt\"" Oct 31 19:33:15.030930 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.030897 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzw49\" (UniqueName: \"kubernetes.io/projected/2de6dc02-e248-424d-81e5-e7754e73b070-kube-api-access-jzw49\") pod \"authorino-56c6c9ddd5-bkkcb\" (UID: \"2de6dc02-e248-424d-81e5-e7754e73b070\") " pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" Oct 31 19:33:15.132009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.131964 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzw49\" (UniqueName: \"kubernetes.io/projected/2de6dc02-e248-424d-81e5-e7754e73b070-kube-api-access-jzw49\") pod \"authorino-56c6c9ddd5-bkkcb\" (UID: \"2de6dc02-e248-424d-81e5-e7754e73b070\") " pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" Oct 31 19:33:15.184761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.184702 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzw49\" (UniqueName: \"kubernetes.io/projected/2de6dc02-e248-424d-81e5-e7754e73b070-kube-api-access-jzw49\") pod \"authorino-56c6c9ddd5-bkkcb\" (UID: \"2de6dc02-e248-424d-81e5-e7754e73b070\") " pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" Oct 31 19:33:15.216378 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.216354 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" Oct 31 19:33:15.569651 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.569619 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bkkcb"] Oct 31 19:33:15.573353 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:33:15.573323 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de6dc02_e248_424d_81e5_e7754e73b070.slice/crio-f99277d339b98679ccee597acf1423222db14b255520bdc08c4167d6acca2c92 WatchSource:0}: Error finding container f99277d339b98679ccee597acf1423222db14b255520bdc08c4167d6acca2c92: Status 404 returned error can't find the container with id f99277d339b98679ccee597acf1423222db14b255520bdc08c4167d6acca2c92 Oct 31 19:33:15.644483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.644449 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" event={"ID":"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca","Type":"ContainerStarted","Data":"fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9"} Oct 31 19:33:15.644657 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.644526 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:15.645666 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.645645 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" event={"ID":"2de6dc02-e248-424d-81e5-e7754e73b070","Type":"ContainerStarted","Data":"f99277d339b98679ccee597acf1423222db14b255520bdc08c4167d6acca2c92"} Oct 31 19:33:15.747869 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:15.747819 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" podStartSLOduration=2.116744934 podStartE2EDuration="5.747805422s" totalImagesPullingTime="3.631060488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:15.747358918 +0000 UTC m=+966.987582051" watchObservedRunningTime="2026-10-31 19:33:15.747805422 +0000 UTC m=+966.988028545" Oct 31 19:33:18.664640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:18.664601 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" event={"ID":"2de6dc02-e248-424d-81e5-e7754e73b070","Type":"ContainerStarted","Data":"2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46"} Oct 31 19:33:18.768663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:18.768610 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" podStartSLOduration=2.235597203 podStartE2EDuration="4.76856934s" totalImagesPullingTime="2.532972137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:18.768177927 +0000 UTC m=+970.008401056" watchObservedRunningTime="2026-10-31 19:33:18.76856934 +0000 UTC m=+970.008792470" Oct 31 19:33:26.651334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:26.651300 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:27.323229 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:33:27.323191 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:33:29.511081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:29.511047 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z27b4"] Oct 31 19:33:29.511537 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:29.511231 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" podUID="91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" containerName="limitador" containerID="cri-o://fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9" gracePeriod=30 Oct 31 19:33:30.048251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.048224 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:30.134074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.133986 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-kube-api-access-mcwks\" (UniqueName: \"kubernetes.io/projected/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-kube-api-access-mcwks\") pod \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " Oct 31 19:33:30.134074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.134053 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-config-file\" (UniqueName: \"kubernetes.io/configmap/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-config-file\") pod \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\" (UID: \"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca\") " Oct 31 19:33:30.134419 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.134395 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-config-file" pod "91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" (UID: "91aa697b-d3ff-4b57-b49e-72e5cf32a9ca"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:30.136249 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.136219 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-kube-api-access-mcwks" pod "91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" (UID: "91aa697b-d3ff-4b57-b49e-72e5cf32a9ca"). InnerVolumeSpecName "kube-api-access-mcwks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:30.234846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.234813 6481 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-config-file\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:30.234846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.234839 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwks\" (UniqueName: \"kubernetes.io/projected/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca-kube-api-access-mcwks\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:30.716761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.716730 6481 generic.go:356] "Generic (PLEG): container finished" podID="91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" containerID="fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9" exitCode=0 Oct 31 19:33:30.717220 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.716767 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" event={"ID":"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca","Type":"ContainerDied","Data":"fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9"} Oct 31 19:33:30.717220 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.716779 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" Oct 31 19:33:30.717220 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.716804 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z27b4" event={"ID":"91aa697b-d3ff-4b57-b49e-72e5cf32a9ca","Type":"ContainerDied","Data":"78c5ed05eff41b91c0446e17e23213aba6c3a5572833d409e2ccb81db54805d1"} Oct 31 19:33:30.717220 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.716827 6481 scope.go:122] "RemoveContainer" containerID="fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9" Oct 31 19:33:30.725550 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.725532 6481 scope.go:122] "RemoveContainer" containerID="fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9" Oct 31 19:33:30.725809 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:33:30.725792 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9\": container with ID starting with fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9 not found: ID does not exist" containerID="fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9" Oct 31 19:33:30.725856 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.725821 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9"} err="failed to get container status \"fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9\": rpc error: code = NotFound desc = could not find container \"fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9\": container with ID starting with fc4156eefb10989e4bc77f5ac35a7e2c547390b72824420d9a2fb87409e7d8c9 not found: ID does not exist" Oct 31 19:33:30.867914 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.867877 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z27b4"] Oct 31 19:33:30.913723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:30.913692 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z27b4"] Oct 31 19:33:32.877121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:32.877086 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" path="/var/lib/kubelet/pods/91aa697b-d3ff-4b57-b49e-72e5cf32a9ca/volumes" Oct 31 19:33:33.546269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.546234 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5"] Oct 31 19:33:33.546587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.546575 6481 cpu_manager.go:414] "containerMap: removing container" podUID="91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" containerName="limitador" Oct 31 19:33:33.546632 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.546589 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" containerName="limitador" Oct 31 19:33:33.546669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.546642 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aa697b-d3ff-4b57-b49e-72e5cf32a9ca" containerName="limitador" Oct 31 19:33:33.550947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.550926 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:33.558886 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.558859 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60fc2851-39a2-4652-bec6-53e5ecc8f4fb-data\") pod \"postgres-7cdbf6dc44-szzc5\" (UID: \"60fc2851-39a2-4652-bec6-53e5ecc8f4fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:33.558982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.558911 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvtg\" (UniqueName: \"kubernetes.io/projected/60fc2851-39a2-4652-bec6-53e5ecc8f4fb-kube-api-access-mjvtg\") pod \"postgres-7cdbf6dc44-szzc5\" (UID: \"60fc2851-39a2-4652-bec6-53e5ecc8f4fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:33.600613 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.600516 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 19:33:33.601693 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.601656 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5"] Oct 31 19:33:33.646557 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.646528 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 19:33:33.646557 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.646547 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5j782\"" Oct 31 19:33:33.646724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.646559 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 19:33:33.660052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.660029 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60fc2851-39a2-4652-bec6-53e5ecc8f4fb-data\") pod \"postgres-7cdbf6dc44-szzc5\" (UID: \"60fc2851-39a2-4652-bec6-53e5ecc8f4fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:33.660156 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.660065 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvtg\" (UniqueName: \"kubernetes.io/projected/60fc2851-39a2-4652-bec6-53e5ecc8f4fb-kube-api-access-mjvtg\") pod \"postgres-7cdbf6dc44-szzc5\" (UID: \"60fc2851-39a2-4652-bec6-53e5ecc8f4fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:33.660420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.660401 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60fc2851-39a2-4652-bec6-53e5ecc8f4fb-data\") pod \"postgres-7cdbf6dc44-szzc5\" (UID: \"60fc2851-39a2-4652-bec6-53e5ecc8f4fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:33.713726 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.713692 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvtg\" (UniqueName: \"kubernetes.io/projected/60fc2851-39a2-4652-bec6-53e5ecc8f4fb-kube-api-access-mjvtg\") pod \"postgres-7cdbf6dc44-szzc5\" (UID: \"60fc2851-39a2-4652-bec6-53e5ecc8f4fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:33.862472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:33.862396 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:34.226990 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:34.226813 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5"] Oct 31 19:33:34.229809 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:33:34.229772 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fc2851_39a2_4652_bec6_53e5ecc8f4fb.slice/crio-09e2c5b638d6f4f6c5b1330be146c3ae1f825d225041ed3c67befdbca1334454 WatchSource:0}: Error finding container 09e2c5b638d6f4f6c5b1330be146c3ae1f825d225041ed3c67befdbca1334454: Status 404 returned error can't find the container with id 09e2c5b638d6f4f6c5b1330be146c3ae1f825d225041ed3c67befdbca1334454 Oct 31 19:33:34.733924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:34.733891 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" event={"ID":"60fc2851-39a2-4652-bec6-53e5ecc8f4fb","Type":"ContainerStarted","Data":"09e2c5b638d6f4f6c5b1330be146c3ae1f825d225041ed3c67befdbca1334454"} Oct 31 19:33:41.763191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:41.763150 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" event={"ID":"60fc2851-39a2-4652-bec6-53e5ecc8f4fb","Type":"ContainerStarted","Data":"48b0d73fede8e8ce76d3f225997f0bd8f233e18a5ae3c8a0f3699b776f71f431"} Oct 31 19:33:41.763613 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:41.763333 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:41.869753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:41.869696 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" podStartSLOduration=1.856960066 podStartE2EDuration="8.869680548s" totalImagesPullingTime="7.012720482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:41.868675662 +0000 UTC m=+993.108898792" watchObservedRunningTime="2026-10-31 19:33:41.869680548 +0000 UTC m=+993.109903676" Oct 31 19:33:47.797164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:47.797129 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-szzc5" Oct 31 19:33:54.749530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.749460 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-n7c7v"] Oct 31 19:33:54.752642 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.752625 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.801307 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.801272 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 19:33:54.815774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.815739 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-metrics-certs\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.815928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.815871 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-webhook-cert\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.815968 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.815924 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27f9\" (UniqueName: \"kubernetes.io/projected/79e062d9-956e-4e6d-90b1-b5438d2daf46-kube-api-access-d27f9\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.824592 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.824566 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 19:33:54.824738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.824645 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rz6ng\"" Oct 31 19:33:54.824804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.824771 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 19:33:54.848562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.848529 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-n7c7v"] Oct 31 19:33:54.917188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.917155 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27f9\" (UniqueName: \"kubernetes.io/projected/79e062d9-956e-4e6d-90b1-b5438d2daf46-kube-api-access-d27f9\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.917400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.917223 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-metrics-certs\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.917400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.917289 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-webhook-cert\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.920186 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.920161 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-metrics-certs\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.920298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.920221 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-webhook-cert\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:54.971077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:54.971016 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27f9\" (UniqueName: \"kubernetes.io/projected/79e062d9-956e-4e6d-90b1-b5438d2daf46-kube-api-access-d27f9\") pod \"maas-controller-579cf7c6d-n7c7v\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:55.065016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.064923 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:55.281033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.280983 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69947f5d84-nnkd9"] Oct 31 19:33:55.284326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.284310 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.320213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.320148 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-webhook-cert\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.320213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.320175 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-metrics-certs\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.320213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.320202 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmxb\" (UniqueName: \"kubernetes.io/projected/3e6e5bb8-1043-4861-8037-a3b781de0cf7-kube-api-access-kpmxb\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.337952 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.337915 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69947f5d84-nnkd9"] Oct 31 19:33:55.421060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.421018 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-webhook-cert\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.421060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.421065 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-metrics-certs\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.421310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.421096 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmxb\" (UniqueName: \"kubernetes.io/projected/3e6e5bb8-1043-4861-8037-a3b781de0cf7-kube-api-access-kpmxb\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.423474 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.423448 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-webhook-cert\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.423580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.423477 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-metrics-certs\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.474260 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.474224 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmxb\" (UniqueName: \"kubernetes.io/projected/3e6e5bb8-1043-4861-8037-a3b781de0cf7-kube-api-access-kpmxb\") pod \"maas-controller-69947f5d84-nnkd9\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.487934 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.487904 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-n7c7v"] Oct 31 19:33:55.491041 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:33:55.491015 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e062d9_956e_4e6d_90b1_b5438d2daf46.slice/crio-32437289a0fc2ad96c90c991d6ec845a1dff227d779c0ec7fb805b2886b39867 WatchSource:0}: Error finding container 32437289a0fc2ad96c90c991d6ec845a1dff227d779c0ec7fb805b2886b39867: Status 404 returned error can't find the container with id 32437289a0fc2ad96c90c991d6ec845a1dff227d779c0ec7fb805b2886b39867 Oct 31 19:33:55.597131 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.597059 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:55.824264 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.824220 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" event={"ID":"79e062d9-956e-4e6d-90b1-b5438d2daf46","Type":"ContainerStarted","Data":"32437289a0fc2ad96c90c991d6ec845a1dff227d779c0ec7fb805b2886b39867"} Oct 31 19:33:55.954968 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:55.954943 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69947f5d84-nnkd9"] Oct 31 19:33:55.957470 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:33:55.957444 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6e5bb8_1043_4861_8037_a3b781de0cf7.slice/crio-2bf130535b4a7ed9645d6eb51600133607781cfe1ce9d1f2ea7ad82f0c59cd1b WatchSource:0}: Error finding container 2bf130535b4a7ed9645d6eb51600133607781cfe1ce9d1f2ea7ad82f0c59cd1b: Status 404 returned error can't find the container with id 2bf130535b4a7ed9645d6eb51600133607781cfe1ce9d1f2ea7ad82f0c59cd1b Oct 31 19:33:56.829546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:56.829510 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-nnkd9" event={"ID":"3e6e5bb8-1043-4861-8037-a3b781de0cf7","Type":"ContainerStarted","Data":"2bf130535b4a7ed9645d6eb51600133607781cfe1ce9d1f2ea7ad82f0c59cd1b"} Oct 31 19:33:57.341526 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:33:57.341460 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:33:59.845124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:59.845089 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-nnkd9" event={"ID":"3e6e5bb8-1043-4861-8037-a3b781de0cf7","Type":"ContainerStarted","Data":"a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35"} Oct 31 19:33:59.845601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:59.845259 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:33:59.846570 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:59.846550 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" event={"ID":"79e062d9-956e-4e6d-90b1-b5438d2daf46","Type":"ContainerStarted","Data":"8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106"} Oct 31 19:33:59.846689 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:59.846677 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:33:59.953150 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:33:59.953098 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-69947f5d84-nnkd9" podStartSLOduration=1.542527805 podStartE2EDuration="4.953085554s" totalImagesPullingTime="3.410557749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:59.952150361 +0000 UTC m=+1011.192373490" watchObservedRunningTime="2026-10-31 19:33:59.953085554 +0000 UTC m=+1011.193308683" Oct 31 19:34:00.067523 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:00.067472 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" podStartSLOduration=2.450073009 podStartE2EDuration="6.067457833s" totalImagesPullingTime="3.617384824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:00.066191362 +0000 UTC m=+1011.306414494" watchObservedRunningTime="2026-10-31 19:34:00.067457833 +0000 UTC m=+1011.307681001" Oct 31 19:34:02.662290 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.662255 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm"] Oct 31 19:34:02.665647 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.665628 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.713965 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.713936 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 19:34:02.714146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.714137 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 19:34:02.714397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.714378 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 19:34:02.760351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.760313 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-k78jw\"" Oct 31 19:34:02.761449 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.761423 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm"] Oct 31 19:34:02.789319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.789276 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd44a116-ca3d-4836-b52d-ed8fe300b085-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.789319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.789312 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.789525 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.789341 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.789525 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.789422 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758x9\" (UniqueName: \"kubernetes.io/projected/dd44a116-ca3d-4836-b52d-ed8fe300b085-kube-api-access-758x9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.890234 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.890191 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd44a116-ca3d-4836-b52d-ed8fe300b085-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.890234 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.890232 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.890457 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.890252 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.890457 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.890280 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758x9\" (UniqueName: \"kubernetes.io/projected/dd44a116-ca3d-4836-b52d-ed8fe300b085-kube-api-access-758x9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.890457 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:02.890336 6481 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 31 19:34:02.890457 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:02.890349 6481 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 31 19:34:02.890457 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:02.890434 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-proxy-source podName:dd44a116-ca3d-4836-b52d-ed8fe300b085 nodeName:}" failed. No retries permitted until 2026-10-31 19:34:03.390415048 +0000 UTC m=+1014.630638174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-x49nm" (UID: "dd44a116-ca3d-4836-b52d-ed8fe300b085") : configmap "usage-tenancy-proxy-source" not found Oct 31 19:34:02.890457 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:02.890453 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44a116-ca3d-4836-b52d-ed8fe300b085-tls-cert podName:dd44a116-ca3d-4836-b52d-ed8fe300b085 nodeName:}" failed. No retries permitted until 2026-10-31 19:34:03.390445836 +0000 UTC m=+1014.630668957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/dd44a116-ca3d-4836-b52d-ed8fe300b085-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-x49nm" (UID: "dd44a116-ca3d-4836-b52d-ed8fe300b085") : secret "usage-logs-tenancy-proxy-tls" not found Oct 31 19:34:02.890957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.890940 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:02.944454 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:02.944374 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758x9\" (UniqueName: \"kubernetes.io/projected/dd44a116-ca3d-4836-b52d-ed8fe300b085-kube-api-access-758x9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:03.395281 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:03.395241 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd44a116-ca3d-4836-b52d-ed8fe300b085-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:03.395488 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:03.395291 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:03.396378 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:03.396353 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dd44a116-ca3d-4836-b52d-ed8fe300b085-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:03.397697 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:03.397675 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd44a116-ca3d-4836-b52d-ed8fe300b085-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x49nm\" (UID: \"dd44a116-ca3d-4836-b52d-ed8fe300b085\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:03.578013 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:03.577966 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:03.941932 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:03.941901 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm"] Oct 31 19:34:03.944834 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:34:03.944807 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd44a116_ca3d_4836_b52d_ed8fe300b085.slice/crio-05a7cd296514e683252e323ead6837e43b00077387c4b5fa4120745269eace55 WatchSource:0}: Error finding container 05a7cd296514e683252e323ead6837e43b00077387c4b5fa4120745269eace55: Status 404 returned error can't find the container with id 05a7cd296514e683252e323ead6837e43b00077387c4b5fa4120745269eace55 Oct 31 19:34:04.870633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:04.870595 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" event={"ID":"dd44a116-ca3d-4836-b52d-ed8fe300b085","Type":"ContainerStarted","Data":"05a7cd296514e683252e323ead6837e43b00077387c4b5fa4120745269eace55"} Oct 31 19:34:10.854240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:10.854207 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 19:34:10.854698 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:10.854320 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:34:11.091041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.090172 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-n7c7v"] Oct 31 19:34:11.091041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.090445 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" podUID="79e062d9-956e-4e6d-90b1-b5438d2daf46" containerName="manager" containerID="cri-o://8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106" gracePeriod=30 Oct 31 19:34:11.354924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.354895 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:34:11.470144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.470106 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-webhook-cert\") pod \"79e062d9-956e-4e6d-90b1-b5438d2daf46\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " Oct 31 19:34:11.470338 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.470209 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e062d9-956e-4e6d-90b1-b5438d2daf46-kube-api-access-d27f9\" (UniqueName: \"kubernetes.io/projected/79e062d9-956e-4e6d-90b1-b5438d2daf46-kube-api-access-d27f9\") pod \"79e062d9-956e-4e6d-90b1-b5438d2daf46\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " Oct 31 19:34:11.470338 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.470265 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-metrics-certs\") pod \"79e062d9-956e-4e6d-90b1-b5438d2daf46\" (UID: \"79e062d9-956e-4e6d-90b1-b5438d2daf46\") " Oct 31 19:34:11.473435 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.473400 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-metrics-certs" pod "79e062d9-956e-4e6d-90b1-b5438d2daf46" (UID: "79e062d9-956e-4e6d-90b1-b5438d2daf46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:11.477527 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.477498 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e062d9-956e-4e6d-90b1-b5438d2daf46-kube-api-access-d27f9" pod "79e062d9-956e-4e6d-90b1-b5438d2daf46" (UID: "79e062d9-956e-4e6d-90b1-b5438d2daf46"). InnerVolumeSpecName "kube-api-access-d27f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:11.477632 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.477541 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-webhook-cert" pod "79e062d9-956e-4e6d-90b1-b5438d2daf46" (UID: "79e062d9-956e-4e6d-90b1-b5438d2daf46"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:11.571391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.571353 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d27f9\" (UniqueName: \"kubernetes.io/projected/79e062d9-956e-4e6d-90b1-b5438d2daf46-kube-api-access-d27f9\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:11.571391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.571394 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:11.571614 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.571409 6481 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e062d9-956e-4e6d-90b1-b5438d2daf46-webhook-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:11.903390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.903309 6481 generic.go:356] "Generic (PLEG): container finished" podID="79e062d9-956e-4e6d-90b1-b5438d2daf46" containerID="8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106" exitCode=0 Oct 31 19:34:11.903390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.903369 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" Oct 31 19:34:11.903888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.903395 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" event={"ID":"79e062d9-956e-4e6d-90b1-b5438d2daf46","Type":"ContainerDied","Data":"8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106"} Oct 31 19:34:11.903888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.903451 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-n7c7v" event={"ID":"79e062d9-956e-4e6d-90b1-b5438d2daf46","Type":"ContainerDied","Data":"32437289a0fc2ad96c90c991d6ec845a1dff227d779c0ec7fb805b2886b39867"} Oct 31 19:34:11.903888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.903473 6481 scope.go:122] "RemoveContainer" containerID="8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106" Oct 31 19:34:11.916203 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.916181 6481 scope.go:122] "RemoveContainer" containerID="8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106" Oct 31 19:34:11.916631 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:11.916605 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106\": container with ID starting with 8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106 not found: ID does not exist" containerID="8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106" Oct 31 19:34:11.916764 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:11.916638 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106"} err="failed to get container status \"8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106\": rpc error: code = NotFound desc = could not find container \"8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106\": container with ID starting with 8000301390be132ecdecca8d0aeecce1ed1abc2692df4dfa44cfa8ba90689106 not found: ID does not exist" Oct 31 19:34:12.068319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:12.068279 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-n7c7v"] Oct 31 19:34:12.082720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:12.082688 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-n7c7v"] Oct 31 19:34:12.879206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:12.879167 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e062d9-956e-4e6d-90b1-b5438d2daf46" path="/var/lib/kubelet/pods/79e062d9-956e-4e6d-90b1-b5438d2daf46/volumes" Oct 31 19:34:17.939637 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:17.939595 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" event={"ID":"dd44a116-ca3d-4836-b52d-ed8fe300b085","Type":"ContainerStarted","Data":"1d728203720a195e53f485632ec09716479627cd4f59f800f5739b5469524389"} Oct 31 19:34:17.940076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:17.939680 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:18.047158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:18.047110 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" podStartSLOduration=2.6414500309999998 podStartE2EDuration="16.047094826s" totalImagesPullingTime="13.405644795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:18.046265226 +0000 UTC m=+1029.286488356" watchObservedRunningTime="2026-10-31 19:34:18.047094826 +0000 UTC m=+1029.287317958" Oct 31 19:34:27.324465 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:27.324427 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:34:28.948375 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:28.948344 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x49nm" Oct 31 19:34:32.729529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.729494 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc"] Oct 31 19:34:32.729928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.729831 6481 cpu_manager.go:414] "containerMap: removing container" podUID="79e062d9-956e-4e6d-90b1-b5438d2daf46" containerName="manager" Oct 31 19:34:32.729928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.729842 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e062d9-956e-4e6d-90b1-b5438d2daf46" containerName="manager" Oct 31 19:34:32.729928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.729895 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e062d9-956e-4e6d-90b1-b5438d2daf46" containerName="manager" Oct 31 19:34:32.735314 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.735297 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.757803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.757778 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-maas-api-tls\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.757907 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.757827 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqh5p\" (UniqueName: \"kubernetes.io/projected/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-kube-api-access-tqh5p\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.757960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.757938 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-openshift-service-ca\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.758007 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.757980 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-metrics-certs\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.833366 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.833334 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 19:34:32.833533 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.833492 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xd7wl\"" Oct 31 19:34:32.833576 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.833541 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 19:34:32.834465 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.834440 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc"] Oct 31 19:34:32.859010 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.858972 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqh5p\" (UniqueName: \"kubernetes.io/projected/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-kube-api-access-tqh5p\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.859125 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.859031 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-openshift-service-ca\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.859125 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.859060 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-metrics-certs\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.859125 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.859097 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-maas-api-tls\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.859733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.859712 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-openshift-service-ca\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.861554 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.861531 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-maas-api-tls\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.861635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.861601 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-metrics-certs\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:32.918872 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:32.918847 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqh5p\" (UniqueName: \"kubernetes.io/projected/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-kube-api-access-tqh5p\") pod \"maas-api-6d669c7fcd-nrbgc\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:33.047795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:33.047708 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:33.411619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:33.411535 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc"] Oct 31 19:34:33.414717 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:34:33.414691 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2dc798_d9c3_4a2a_865c_106f6e26b76b.slice/crio-7c1c434dba4ba15f6a9dc89914094967ec95d723d18dba564fe8c56db7dcf7bb WatchSource:0}: Error finding container 7c1c434dba4ba15f6a9dc89914094967ec95d723d18dba564fe8c56db7dcf7bb: Status 404 returned error can't find the container with id 7c1c434dba4ba15f6a9dc89914094967ec95d723d18dba564fe8c56db7dcf7bb Oct 31 19:34:34.002264 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:34.002229 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" event={"ID":"fb2dc798-d9c3-4a2a-865c-106f6e26b76b","Type":"ContainerStarted","Data":"7c1c434dba4ba15f6a9dc89914094967ec95d723d18dba564fe8c56db7dcf7bb"} Oct 31 19:34:36.011905 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:36.011870 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" event={"ID":"fb2dc798-d9c3-4a2a-865c-106f6e26b76b","Type":"ContainerStarted","Data":"fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3"} Oct 31 19:34:36.012311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:36.012031 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:36.120650 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:36.120593 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" podStartSLOduration=2.041995665 podStartE2EDuration="4.120576226s" totalImagesPullingTime="2.078580561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:36.120534871 +0000 UTC m=+1047.360758013" watchObservedRunningTime="2026-10-31 19:34:36.120576226 +0000 UTC m=+1047.360799356" Oct 31 19:34:42.021434 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:42.021405 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:43.733057 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:43.733021 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bb479dcdb-n84h4"] Oct 31 19:34:43.876061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:43.876026 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bb479dcdb-n84h4"] Oct 31 19:34:43.876213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:43.876086 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:43.924838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:43.924814 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 19:34:43.947288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:43.947260 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lzz\" (UniqueName: \"kubernetes.io/projected/d39947ab-1440-45f9-943d-5c33cd271148-kube-api-access-n4lzz\") pod \"authorino-bb479dcdb-n84h4\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:43.947406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:43.947384 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d39947ab-1440-45f9-943d-5c33cd271148-tls-cert\") pod \"authorino-bb479dcdb-n84h4\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:44.048796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:44.048719 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d39947ab-1440-45f9-943d-5c33cd271148-tls-cert\") pod \"authorino-bb479dcdb-n84h4\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:44.048796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:44.048784 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lzz\" (UniqueName: \"kubernetes.io/projected/d39947ab-1440-45f9-943d-5c33cd271148-kube-api-access-n4lzz\") pod \"authorino-bb479dcdb-n84h4\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:44.051250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:44.051218 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d39947ab-1440-45f9-943d-5c33cd271148-tls-cert\") pod \"authorino-bb479dcdb-n84h4\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:44.104286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:44.104257 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lzz\" (UniqueName: \"kubernetes.io/projected/d39947ab-1440-45f9-943d-5c33cd271148-kube-api-access-n4lzz\") pod \"authorino-bb479dcdb-n84h4\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:44.206601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:44.206569 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:34:44.596069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:44.595907 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bb479dcdb-n84h4"] Oct 31 19:34:44.598800 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:34:44.598771 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd39947ab_1440_45f9_943d_5c33cd271148.slice/crio-8b8a5a29e8558aae93cb57cabec8f3fae15634c138de22d4525cb6892c216a88 WatchSource:0}: Error finding container 8b8a5a29e8558aae93cb57cabec8f3fae15634c138de22d4525cb6892c216a88: Status 404 returned error can't find the container with id 8b8a5a29e8558aae93cb57cabec8f3fae15634c138de22d4525cb6892c216a88 Oct 31 19:34:45.051673 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:45.051632 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bb479dcdb-n84h4" event={"ID":"d39947ab-1440-45f9-943d-5c33cd271148","Type":"ContainerStarted","Data":"8b8a5a29e8558aae93cb57cabec8f3fae15634c138de22d4525cb6892c216a88"} Oct 31 19:34:47.068316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.068277 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bb479dcdb-n84h4" event={"ID":"d39947ab-1440-45f9-943d-5c33cd271148","Type":"ContainerStarted","Data":"8be3148834d006a55d39b4ea873acd88a1e71828a60efded36f2e100128a9fc5"} Oct 31 19:34:47.175095 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.175036 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-bb479dcdb-n84h4" podStartSLOduration=2.51304014 podStartE2EDuration="4.175017224s" totalImagesPullingTime="1.661977084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:47.174860403 +0000 UTC m=+1058.415083533" watchObservedRunningTime="2026-10-31 19:34:47.175017224 +0000 UTC m=+1058.415240358" Oct 31 19:34:47.222071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.222035 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bkkcb"] Oct 31 19:34:47.222313 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.222285 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" podUID="2de6dc02-e248-424d-81e5-e7754e73b070" containerName="authorino" containerID="cri-o://2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46" gracePeriod=30 Oct 31 19:34:47.509828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.509795 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" Oct 31 19:34:47.587685 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.587658 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de6dc02-e248-424d-81e5-e7754e73b070-kube-api-access-jzw49\" (UniqueName: \"kubernetes.io/projected/2de6dc02-e248-424d-81e5-e7754e73b070-kube-api-access-jzw49\") pod \"2de6dc02-e248-424d-81e5-e7754e73b070\" (UID: \"2de6dc02-e248-424d-81e5-e7754e73b070\") " Oct 31 19:34:47.589631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.589606 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de6dc02-e248-424d-81e5-e7754e73b070-kube-api-access-jzw49" pod "2de6dc02-e248-424d-81e5-e7754e73b070" (UID: "2de6dc02-e248-424d-81e5-e7754e73b070"). InnerVolumeSpecName "kube-api-access-jzw49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:47.688420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:47.688349 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzw49\" (UniqueName: \"kubernetes.io/projected/2de6dc02-e248-424d-81e5-e7754e73b070-kube-api-access-jzw49\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:48.073568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.073533 6481 generic.go:356] "Generic (PLEG): container finished" podID="2de6dc02-e248-424d-81e5-e7754e73b070" containerID="2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46" exitCode=0 Oct 31 19:34:48.074034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.073613 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" event={"ID":"2de6dc02-e248-424d-81e5-e7754e73b070","Type":"ContainerDied","Data":"2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46"} Oct 31 19:34:48.074034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.073635 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" Oct 31 19:34:48.074034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.073662 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bkkcb" event={"ID":"2de6dc02-e248-424d-81e5-e7754e73b070","Type":"ContainerDied","Data":"f99277d339b98679ccee597acf1423222db14b255520bdc08c4167d6acca2c92"} Oct 31 19:34:48.074034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.073681 6481 scope.go:122] "RemoveContainer" containerID="2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46" Oct 31 19:34:48.082165 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.082143 6481 scope.go:122] "RemoveContainer" containerID="2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46" Oct 31 19:34:48.082482 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:48.082454 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46\": container with ID starting with 2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46 not found: ID does not exist" containerID="2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46" Oct 31 19:34:48.082576 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.082495 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46"} err="failed to get container status \"2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46\": rpc error: code = NotFound desc = could not find container \"2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46\": container with ID starting with 2d7d72dbc91a0266d60cabd8399d1871622ec2997507ec353f2b1775b7a4fa46 not found: ID does not exist" Oct 31 19:34:48.224736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.224705 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bkkcb"] Oct 31 19:34:48.228469 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.228445 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bkkcb"] Oct 31 19:34:48.877408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:48.877379 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de6dc02-e248-424d-81e5-e7754e73b070" path="/var/lib/kubelet/pods/2de6dc02-e248-424d-81e5-e7754e73b070/volumes" Oct 31 19:34:53.389476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.389440 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc"] Oct 31 19:34:53.389900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.389632 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" podUID="fb2dc798-d9c3-4a2a-865c-106f6e26b76b" containerName="maas-api" containerID="cri-o://fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3" gracePeriod=30 Oct 31 19:34:53.637778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.637746 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:53.741179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.741151 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-kube-api-access-tqh5p\" (UniqueName: \"kubernetes.io/projected/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-kube-api-access-tqh5p\") pod \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " Oct 31 19:34:53.741369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.741199 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-metrics-certs\") pod \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " Oct 31 19:34:53.741369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.741255 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-maas-api-tls\") pod \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " Oct 31 19:34:53.741369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.741288 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-openshift-service-ca\") pod \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\" (UID: \"fb2dc798-d9c3-4a2a-865c-106f6e26b76b\") " Oct 31 19:34:53.741677 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.741651 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-openshift-service-ca" pod "fb2dc798-d9c3-4a2a-865c-106f6e26b76b" (UID: "fb2dc798-d9c3-4a2a-865c-106f6e26b76b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:53.743507 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.743474 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-metrics-certs" pod "fb2dc798-d9c3-4a2a-865c-106f6e26b76b" (UID: "fb2dc798-d9c3-4a2a-865c-106f6e26b76b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:53.743618 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.743532 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-kube-api-access-tqh5p" pod "fb2dc798-d9c3-4a2a-865c-106f6e26b76b" (UID: "fb2dc798-d9c3-4a2a-865c-106f6e26b76b"). InnerVolumeSpecName "kube-api-access-tqh5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:53.743618 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.743591 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-maas-api-tls" pod "fb2dc798-d9c3-4a2a-865c-106f6e26b76b" (UID: "fb2dc798-d9c3-4a2a-865c-106f6e26b76b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:53.842724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.842691 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:53.842724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.842718 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:53.842724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.842728 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:53.842944 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:53.842738 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqh5p\" (UniqueName: \"kubernetes.io/projected/fb2dc798-d9c3-4a2a-865c-106f6e26b76b-kube-api-access-tqh5p\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:54.101724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.101644 6481 generic.go:356] "Generic (PLEG): container finished" podID="fb2dc798-d9c3-4a2a-865c-106f6e26b76b" containerID="fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3" exitCode=0 Oct 31 19:34:54.101724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.101716 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" Oct 31 19:34:54.101897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.101762 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" event={"ID":"fb2dc798-d9c3-4a2a-865c-106f6e26b76b","Type":"ContainerDied","Data":"fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3"} Oct 31 19:34:54.101897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.101829 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc" event={"ID":"fb2dc798-d9c3-4a2a-865c-106f6e26b76b","Type":"ContainerDied","Data":"7c1c434dba4ba15f6a9dc89914094967ec95d723d18dba564fe8c56db7dcf7bb"} Oct 31 19:34:54.101897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.101856 6481 scope.go:122] "RemoveContainer" containerID="fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3" Oct 31 19:34:54.110837 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.110813 6481 scope.go:122] "RemoveContainer" containerID="fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3" Oct 31 19:34:54.111112 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:54.111096 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3\": container with ID starting with fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3 not found: ID does not exist" containerID="fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3" Oct 31 19:34:54.111175 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.111121 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3"} err="failed to get container status \"fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3\": rpc error: code = NotFound desc = could not find container \"fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3\": container with ID starting with fb297cf8711cacb36fa8db1923876c4522fe51c9fd12b5d9e07bbfc197a368d3 not found: ID does not exist" Oct 31 19:34:54.253901 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.253869 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc"] Oct 31 19:34:54.259390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.259364 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nrbgc"] Oct 31 19:34:54.877972 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:34:54.877929 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2dc798-d9c3-4a2a-865c-106f6e26b76b" path="/var/lib/kubelet/pods/fb2dc798-d9c3-4a2a-865c-106f6e26b76b/volumes" Oct 31 19:34:57.323877 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:34:57.323841 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:35:27.322949 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:35:27.322860 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:35:27.509023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.508975 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg"] Oct 31 19:35:27.509332 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.509320 6481 cpu_manager.go:414] "containerMap: removing container" podUID="2de6dc02-e248-424d-81e5-e7754e73b070" containerName="authorino" Oct 31 19:35:27.509374 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.509333 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de6dc02-e248-424d-81e5-e7754e73b070" containerName="authorino" Oct 31 19:35:27.509374 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.509347 6481 cpu_manager.go:414] "containerMap: removing container" podUID="fb2dc798-d9c3-4a2a-865c-106f6e26b76b" containerName="maas-api" Oct 31 19:35:27.509374 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.509352 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2dc798-d9c3-4a2a-865c-106f6e26b76b" containerName="maas-api" Oct 31 19:35:27.509467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.509407 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de6dc02-e248-424d-81e5-e7754e73b070" containerName="authorino" Oct 31 19:35:27.509467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.509415 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2dc798-d9c3-4a2a-865c-106f6e26b76b" containerName="maas-api" Oct 31 19:35:27.512187 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.512172 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532604 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532760 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532662 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532760 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532679 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532760 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532707 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532860 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532813 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532849 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532944 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532911 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.532944 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.532935 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.533071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.533053 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shghl\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-kube-api-access-shghl\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.556494 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.556467 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn"] Oct 31 19:35:27.559565 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.559550 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.601773 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.601695 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5cqm6\"" Oct 31 19:35:27.602934 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.602913 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg"] Oct 31 19:35:27.634198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634164 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.634412 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634387 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.634528 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634449 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.634528 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634481 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.634528 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634524 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634546 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634577 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634602 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634641 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634667 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634691 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634714 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.634751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634738 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f846q\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-kube-api-access-f846q\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.635158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634790 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.635158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634822 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.635158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.634948 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.635158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.635052 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.635407 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.635241 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.635407 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.635248 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shghl\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-kube-api-access-shghl\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.635503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.635478 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.635767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.635736 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.636296 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.635989 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.636296 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.636074 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.636296 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.636130 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn"] Oct 31 19:35:27.638396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.638343 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.638930 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.638908 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.692562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.692534 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.692679 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.692610 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shghl\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-kube-api-access-shghl\") pod \"maas-default-gateway-openshift-default-58d898bd55-sf8qg\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.736518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736479 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736517 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736545 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736560 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f846q\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-kube-api-access-f846q\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736589 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736615 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736648 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736707 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.736758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.736736 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.737115 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.737080 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.737202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.737180 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.737258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.737235 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.737301 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.737264 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.737480 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.737464 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.739446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.739427 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.739703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.739685 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.792293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.792251 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f846q\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-kube-api-access-f846q\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.792476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.792364 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-9qngn\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:27.826084 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.826055 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:27.870923 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:27.870852 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:28.242155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:28.242127 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg"] Oct 31 19:35:28.276548 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:35:28.276517 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bffa3ac_6da4_442b_922a_6ef7cc1a5f31.slice/crio-d7591ec537901f4ac2114534ec6298c1a74417da64c6a9f6d4a7b0f467cb4231 WatchSource:0}: Error finding container d7591ec537901f4ac2114534ec6298c1a74417da64c6a9f6d4a7b0f467cb4231: Status 404 returned error can't find the container with id d7591ec537901f4ac2114534ec6298c1a74417da64c6a9f6d4a7b0f467cb4231 Oct 31 19:35:28.287989 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:28.287942 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn"] Oct 31 19:35:28.290249 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:35:28.290225 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabab2ca3_aa23_4dcf_ab92_d01fd1002309.slice/crio-19ca592d0fa56f8c2d30ec62d331a6b3435fef2afcb27b9f1437870685487395 WatchSource:0}: Error finding container 19ca592d0fa56f8c2d30ec62d331a6b3435fef2afcb27b9f1437870685487395: Status 404 returned error can't find the container with id 19ca592d0fa56f8c2d30ec62d331a6b3435fef2afcb27b9f1437870685487395 Oct 31 19:35:29.250083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:29.250044 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" event={"ID":"abab2ca3-aa23-4dcf-ab92-d01fd1002309","Type":"ContainerStarted","Data":"19ca592d0fa56f8c2d30ec62d331a6b3435fef2afcb27b9f1437870685487395"} Oct 31 19:35:29.251215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:29.251187 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" event={"ID":"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31","Type":"ContainerStarted","Data":"d7591ec537901f4ac2114534ec6298c1a74417da64c6a9f6d4a7b0f467cb4231"} Oct 31 19:35:32.464216 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:32.464175 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:32.464522 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:32.464264 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:32.464522 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:32.464305 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:32.473613 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:32.473577 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:32.473726 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:32.473647 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:32.473726 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:32.473684 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:33.291017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.287803 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" event={"ID":"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31","Type":"ContainerStarted","Data":"b3d22035b8ff86bfded35602808ec67736d760f116defd646a3574ecbe64d650"} Oct 31 19:35:33.316561 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.316514 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" event={"ID":"abab2ca3-aa23-4dcf-ab92-d01fd1002309","Type":"ContainerStarted","Data":"ac9d0933e77d47dcb2c650191b79db1a6e7b4bf5b68b51e42a9603c61c2e19de"} Oct 31 19:35:33.454224 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.454175 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" podStartSLOduration=2.26927122 podStartE2EDuration="6.454162139s" totalImagesPullingTime="4.184890919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:33.4075845 +0000 UTC m=+1104.647807627" watchObservedRunningTime="2026-10-31 19:35:33.454162139 +0000 UTC m=+1104.694385268" Oct 31 19:35:33.518905 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.518846 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" podStartSLOduration=2.337991405 podStartE2EDuration="6.518830082s" totalImagesPullingTime="4.180838677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:33.517957925 +0000 UTC m=+1104.758181056" watchObservedRunningTime="2026-10-31 19:35:33.518830082 +0000 UTC m=+1104.759053217" Oct 31 19:35:33.827039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.827007 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:33.828410 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.828378 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 19:35:33.828536 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.828434 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 19:35:33.874787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.874743 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:33.875183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.875103 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-9qngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 19:35:33.875355 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:33.875334 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 19:35:34.826754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:34.826712 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 19:35:34.827267 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:34.826772 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 19:35:34.889016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:34.888384 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-9qngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 19:35:34.889016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:34.888441 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 19:35:35.827130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:35.827090 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 19:35:35.827512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:35.827143 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 19:35:35.874468 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:35.874416 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-9qngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 19:35:35.874799 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:35.874757 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 19:35:36.827464 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:36.827423 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 19:35:36.827854 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:36.827489 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 19:35:36.909017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:36.907351 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-9qngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 19:35:36.909017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:36.907404 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 19:35:37.826721 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:37.826685 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 19:35:37.826877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:37.826750 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 19:35:37.871468 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:37.871431 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-9qngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 19:35:37.872806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:37.872773 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 19:35:38.831061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:38.831027 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:38.831334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:38.831306 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:38.831986 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:38.831968 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:35:38.877030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:38.876985 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:38.877392 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:38.877263 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:38.877960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:38.877941 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:35:42.604839 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.604805 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv"] Oct 31 19:35:42.609596 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.609578 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.714773 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.714732 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.714926 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.714777 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.714926 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.714801 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5451f44e-6729-4371-bd5c-2c6d991ab356-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.715025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.714923 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.715025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.715016 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.715093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.715043 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.715093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.715061 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.715177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.715151 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.715215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.715203 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q557m\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-kube-api-access-q557m\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.814369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.814336 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv"] Oct 31 19:35:42.816587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.816559 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.816736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.816602 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q557m\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-kube-api-access-q557m\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.816736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.816649 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.816736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.816672 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.816736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.816694 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5451f44e-6729-4371-bd5c-2c6d991ab356-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.816944 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.816735 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.816934 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.817012 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.817026 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.817043 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817246 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.817176 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817435 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.817390 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.817502 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.817603 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.817578 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5451f44e-6729-4371-bd5c-2c6d991ab356-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.819367 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.819341 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.819557 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.819540 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.870386 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.870311 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.870531 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.870419 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q557m\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-kube-api-access-q557m\") pod \"maas-default-gateway-openshift-default-58d898bd55-dxgfv\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:42.922793 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:42.922754 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:43.031768 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.031736 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5"] Oct 31 19:35:43.036306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.036287 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.086944 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.086914 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l8x7h\"" Oct 31 19:35:43.087130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.087016 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 19:35:43.087130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.087019 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 19:35:43.087130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.087078 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 19:35:43.089592 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.089568 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5"] Oct 31 19:35:43.119978 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.119952 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.120102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.119990 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x6zd\" (UniqueName: \"kubernetes.io/projected/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-kube-api-access-6x6zd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.120143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.120112 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.120177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.120168 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.120244 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.120230 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.120311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.120296 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.220969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.220940 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221013 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221041 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221070 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221184 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221269 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x6zd\" (UniqueName: \"kubernetes.io/projected/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-kube-api-access-6x6zd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221454 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221402 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221454 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221429 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.221636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.221522 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.223340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.223324 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.223518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.223497 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.274955 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.274925 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x6zd\" (UniqueName: \"kubernetes.io/projected/038fbde1-ebd7-45b2-a2e2-acd0c92e1870-kube-api-access-6x6zd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-hcnz5\" (UID: \"038fbde1-ebd7-45b2-a2e2-acd0c92e1870\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.335743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.335715 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv"] Oct 31 19:35:43.338211 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:35:43.338185 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5451f44e_6729_4371_bd5c_2c6d991ab356.slice/crio-f78ecf3a5d2f924d2d7a62714aeea3d84ede4c2f83085f4d761e2f485f936a2c WatchSource:0}: Error finding container f78ecf3a5d2f924d2d7a62714aeea3d84ede4c2f83085f4d761e2f485f936a2c: Status 404 returned error can't find the container with id f78ecf3a5d2f924d2d7a62714aeea3d84ede4c2f83085f4d761e2f485f936a2c Oct 31 19:35:43.340216 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.340188 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:43.340262 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.340246 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:43.340298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.340274 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:43.348928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.348906 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:43.361300 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.361276 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" event={"ID":"5451f44e-6729-4371-bd5c-2c6d991ab356","Type":"ContainerStarted","Data":"f78ecf3a5d2f924d2d7a62714aeea3d84ede4c2f83085f4d761e2f485f936a2c"} Oct 31 19:35:43.721104 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:43.721074 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5"] Oct 31 19:35:43.721489 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:35:43.721465 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038fbde1_ebd7_45b2_a2e2_acd0c92e1870.slice/crio-221352a8a39f82984069808bf31ac7b4c9295239e28e5717e5a66e9646717bb0 WatchSource:0}: Error finding container 221352a8a39f82984069808bf31ac7b4c9295239e28e5717e5a66e9646717bb0: Status 404 returned error can't find the container with id 221352a8a39f82984069808bf31ac7b4c9295239e28e5717e5a66e9646717bb0 Oct 31 19:35:44.394286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:44.394242 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" event={"ID":"5451f44e-6729-4371-bd5c-2c6d991ab356","Type":"ContainerStarted","Data":"6795cd00de576e3bb54322853fe22d2855b073741fb069f75339fe3a416dd220"} Oct 31 19:35:44.395538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:44.395511 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" event={"ID":"038fbde1-ebd7-45b2-a2e2-acd0c92e1870","Type":"ContainerStarted","Data":"221352a8a39f82984069808bf31ac7b4c9295239e28e5717e5a66e9646717bb0"} Oct 31 19:35:44.501649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:44.501596 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podStartSLOduration=2.50158261 podStartE2EDuration="2.50158261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:44.500372051 +0000 UTC m=+1115.740595184" watchObservedRunningTime="2026-10-31 19:35:44.50158261 +0000 UTC m=+1115.741805787" Oct 31 19:35:44.923955 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:44.923903 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:44.926375 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:44.925472 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dxgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 19:35:44.926375 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:44.925528 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 19:35:45.923483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:45.923453 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dxgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 19:35:45.923641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:45.923504 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 19:35:46.407524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:46.407485 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" event={"ID":"038fbde1-ebd7-45b2-a2e2-acd0c92e1870","Type":"ContainerStarted","Data":"6649b1beddb4b3d5619bc4dede74866236e439ea00cb0900612e392e1e29f760"} Oct 31 19:35:46.514279 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:46.514216 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" podStartSLOduration=2.301023644 podStartE2EDuration="4.514196855s" totalImagesPullingTime="2.213173211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:46.513704279 +0000 UTC m=+1117.753927421" watchObservedRunningTime="2026-10-31 19:35:46.514196855 +0000 UTC m=+1117.754419987" Oct 31 19:35:46.929326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:46.929288 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dxgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 19:35:46.929633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:46.929588 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 19:35:47.349754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:47.349696 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:47.923686 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:47.923649 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dxgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 19:35:47.924140 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:47.923702 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 19:35:48.923895 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:48.923863 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dxgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 19:35:48.924292 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:48.923913 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 19:35:49.927595 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:49.927557 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:49.928007 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:49.927832 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:49.928664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:49.928644 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:35:53.349653 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:53.349613 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:53.362132 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:53.362105 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:53.448095 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:53.448067 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-hcnz5" Oct 31 19:35:57.323955 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:35:57.323915 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:35:57.424543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.424440 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2"] Oct 31 19:35:57.429541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.429526 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.525424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.525392 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 19:35:57.526735 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.526709 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2"] Oct 31 19:35:57.582550 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.582458 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vr2\" (UniqueName: \"kubernetes.io/projected/9de1e1d1-192a-488e-8f42-e013800a9dcf-kube-api-access-g2vr2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.582550 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.582521 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.582765 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.582620 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.582765 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.582651 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de1e1d1-192a-488e-8f42-e013800a9dcf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.582765 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.582679 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.582931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.582818 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683335 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vr2\" (UniqueName: \"kubernetes.io/projected/9de1e1d1-192a-488e-8f42-e013800a9dcf-kube-api-access-g2vr2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683421 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683454 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683471 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de1e1d1-192a-488e-8f42-e013800a9dcf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683492 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683532 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683831 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683913 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683894 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.683971 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.683952 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.685909 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.685887 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de1e1d1-192a-488e-8f42-e013800a9dcf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.686153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.686134 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de1e1d1-192a-488e-8f42-e013800a9dcf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.720244 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.720212 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6"] Oct 31 19:35:57.723618 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.723593 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.739900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.739871 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vr2\" (UniqueName: \"kubernetes.io/projected/9de1e1d1-192a-488e-8f42-e013800a9dcf-kube-api-access-g2vr2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2\" (UID: \"9de1e1d1-192a-488e-8f42-e013800a9dcf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.741739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.741714 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:57.778785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.778718 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6"] Oct 31 19:35:57.885347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885266 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885341 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885399 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7838e08-5257-4a6c-8861-4331b2b2fd98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885471 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885510 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885566 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggb5b\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-kube-api-access-ggb5b\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885595 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885671 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.885749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.885696 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.986937 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.986900 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggb5b\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-kube-api-access-ggb5b\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.986942 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987061 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987099 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987135 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987162 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987223 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7838e08-5257-4a6c-8861-4331b2b2fd98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987299 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987324 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987549 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987437 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987651 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987543 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987705 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987646 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987745 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987703 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.987947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.987926 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7838e08-5257-4a6c-8861-4331b2b2fd98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.990050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.990023 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:57.990148 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:57.990134 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:58.040673 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.040643 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:58.040817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.040764 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggb5b\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-kube-api-access-ggb5b\") pod \"maas-default-gateway-openshift-default-58d898bd55-gvdt6\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:58.118443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.118267 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2"] Oct 31 19:35:58.121210 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:35:58.121180 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de1e1d1_192a_488e_8f42_e013800a9dcf.slice/crio-f2d866edd302f8e7004f49a96c271cf2a1826e03e945e9bf8eef6828d45b1b89 WatchSource:0}: Error finding container f2d866edd302f8e7004f49a96c271cf2a1826e03e945e9bf8eef6828d45b1b89: Status 404 returned error can't find the container with id f2d866edd302f8e7004f49a96c271cf2a1826e03e945e9bf8eef6828d45b1b89 Oct 31 19:35:58.336337 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.336299 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:35:58.461055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.461020 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" event={"ID":"9de1e1d1-192a-488e-8f42-e013800a9dcf","Type":"ContainerStarted","Data":"f2d866edd302f8e7004f49a96c271cf2a1826e03e945e9bf8eef6828d45b1b89"} Oct 31 19:35:58.748269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.748247 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6"] Oct 31 19:35:58.752199 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:35:58.752164 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7838e08_5257_4a6c_8861_4331b2b2fd98.slice/crio-a7842fbc010b205b701f43bf8d988cf45aceb83799ed1cdfc126446ccaeabcf5 WatchSource:0}: Error finding container a7842fbc010b205b701f43bf8d988cf45aceb83799ed1cdfc126446ccaeabcf5: Status 404 returned error can't find the container with id a7842fbc010b205b701f43bf8d988cf45aceb83799ed1cdfc126446ccaeabcf5 Oct 31 19:35:58.754693 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.754664 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:58.754806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.754737 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:58.754806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:58.754790 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:35:59.487018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:59.484710 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" event={"ID":"d7838e08-5257-4a6c-8861-4331b2b2fd98","Type":"ContainerStarted","Data":"af7d3ea68125aca848a35ba2476f8f47af50b8e657cffd9da1b169dd14c62c6a"} Oct 31 19:35:59.487018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:59.484780 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" event={"ID":"d7838e08-5257-4a6c-8861-4331b2b2fd98","Type":"ContainerStarted","Data":"a7842fbc010b205b701f43bf8d988cf45aceb83799ed1cdfc126446ccaeabcf5"} Oct 31 19:35:59.491860 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:59.491825 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" event={"ID":"9de1e1d1-192a-488e-8f42-e013800a9dcf","Type":"ContainerStarted","Data":"ab3db0fab08edc7f0da0a14e6d843a0f3ec9647750d2537a03cffeee8c57d48e"} Oct 31 19:35:59.650781 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:59.650721 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podStartSLOduration=2.650707045 podStartE2EDuration="2.650707045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:59.649166853 +0000 UTC m=+1130.889389982" watchObservedRunningTime="2026-10-31 19:35:59.650707045 +0000 UTC m=+1130.890930173" Oct 31 19:35:59.742666 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:59.742583 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:35:59.756583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:35:59.756524 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" podStartSLOduration=2.15939843 podStartE2EDuration="2.7565097s" totalImagesPullingTime="597.11127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:59.755655456 +0000 UTC m=+1130.995878588" watchObservedRunningTime="2026-10-31 19:35:59.7565097 +0000 UTC m=+1130.996732826" Oct 31 19:36:00.337243 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:00.337206 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:36:00.338678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:00.338652 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:36:00.338816 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:00.338703 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 19:36:01.337105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:01.337067 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:36:01.337606 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:01.337131 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 19:36:02.336749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:02.336716 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:36:02.343185 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:02.343133 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 19:36:03.337230 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:03.337189 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:36:03.337394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:03.337256 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 19:36:04.337144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.337104 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 19:36:04.337711 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.337164 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 19:36:04.892572 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.892533 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn"] Oct 31 19:36:04.900467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.900443 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:04.978243 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.978213 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 19:36:04.999289 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.999257 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hkq\" (UniqueName: \"kubernetes.io/projected/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-kube-api-access-k5hkq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:04.999433 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.999300 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:04.999433 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.999422 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:04.999511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.999443 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:04.999511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.999463 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:04.999511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:04.999488 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.001938 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.001915 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn"] Oct 31 19:36:05.100792 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.100756 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.100960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.100836 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.100960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.100860 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.100960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.100878 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.100960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.100903 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.101200 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.101062 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hkq\" (UniqueName: \"kubernetes.io/projected/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-kube-api-access-k5hkq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.101376 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.101348 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.101467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.101371 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.101467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.101426 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.103398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.103375 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.103576 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.103559 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.155143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.155075 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hkq\" (UniqueName: \"kubernetes.io/projected/7e7c0d83-8f3b-4130-9469-1d70e37eb9b3-kube-api-access-k5hkq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn\" (UID: \"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.213287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.213241 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:05.579787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:05.579760 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn"] Oct 31 19:36:05.582408 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:36:05.582380 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7c0d83_8f3b_4130_9469_1d70e37eb9b3.slice/crio-30f8ef23e7d1a5adb226f06bcc70b7cdc252574d10397530849038c940f10597 WatchSource:0}: Error finding container 30f8ef23e7d1a5adb226f06bcc70b7cdc252574d10397530849038c940f10597: Status 404 returned error can't find the container with id 30f8ef23e7d1a5adb226f06bcc70b7cdc252574d10397530849038c940f10597 Oct 31 19:36:06.338124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.338079 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:36:06.338301 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.338148 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:06.411258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.411224 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:36:06.411637 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.411610 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:36:06.413044 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.413021 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:36:06.541179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.541134 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" event={"ID":"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3","Type":"ContainerStarted","Data":"4f9f94e186e141ee5a6341b6d253fd3926476e6ae06576f05d8b0b6d18bf00e4"} Oct 31 19:36:06.541361 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.541199 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" event={"ID":"7e7c0d83-8f3b-4130-9469-1d70e37eb9b3","Type":"ContainerStarted","Data":"30f8ef23e7d1a5adb226f06bcc70b7cdc252574d10397530849038c940f10597"} Oct 31 19:36:06.693428 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:06.693370 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" podStartSLOduration=2.08999951 podStartE2EDuration="2.693352915s" totalImagesPullingTime="603.353405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:06.691842833 +0000 UTC m=+1137.932065962" watchObservedRunningTime="2026-10-31 19:36:06.693352915 +0000 UTC m=+1137.933576045" Oct 31 19:36:07.214546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:07.214500 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:07.745601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:07.745570 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:36:07.789629 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:07.789345 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:36:08.596019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:08.595342 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" Oct 31 19:36:14.517018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.516493 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n"] Oct 31 19:36:14.522541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.522509 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.620713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.620677 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n"] Oct 31 19:36:14.620901 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.620822 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 19:36:14.633846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.632957 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.633846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.633082 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tlf\" (UniqueName: \"kubernetes.io/projected/6f8e9398-998f-42be-8dee-f0e169426697-kube-api-access-v6tlf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.633846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.633119 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.633846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.633198 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.633846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.633270 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8e9398-998f-42be-8dee-f0e169426697-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.633846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.633667 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.735372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.735327 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8e9398-998f-42be-8dee-f0e169426697-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.735586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.735391 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.735586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.735458 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.735586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.735520 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tlf\" (UniqueName: \"kubernetes.io/projected/6f8e9398-998f-42be-8dee-f0e169426697-kube-api-access-v6tlf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.735586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.735552 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.735903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.735618 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.736105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.736079 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.736381 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.736351 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.736649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.736622 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.738701 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.738649 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8e9398-998f-42be-8dee-f0e169426697-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.745723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.745685 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8e9398-998f-42be-8dee-f0e169426697-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.801682 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.801600 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tlf\" (UniqueName: \"kubernetes.io/projected/6f8e9398-998f-42be-8dee-f0e169426697-kube-api-access-v6tlf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q272n\" (UID: \"6f8e9398-998f-42be-8dee-f0e169426697\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:14.842117 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:14.842079 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:15.238744 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:15.236392 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:15.291864 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:15.291588 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:15.301610 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:15.301575 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n"] Oct 31 19:36:15.351080 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:36:15.350098 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8e9398_998f_42be_8dee_f0e169426697.slice/crio-c288f2f035acea5a01754eeaa192162ade62b07aef420cc24901cc010b8711e8 WatchSource:0}: Error finding container c288f2f035acea5a01754eeaa192162ade62b07aef420cc24901cc010b8711e8: Status 404 returned error can't find the container with id c288f2f035acea5a01754eeaa192162ade62b07aef420cc24901cc010b8711e8 Oct 31 19:36:15.712192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:15.712153 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" event={"ID":"6f8e9398-998f-42be-8dee-f0e169426697","Type":"ContainerStarted","Data":"c288f2f035acea5a01754eeaa192162ade62b07aef420cc24901cc010b8711e8"} Oct 31 19:36:15.772018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:15.762331 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" Oct 31 19:36:16.704154 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:16.704060 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" event={"ID":"6f8e9398-998f-42be-8dee-f0e169426697","Type":"ContainerStarted","Data":"eed1b126afd53bc0a7e6f6913b4126577c8838542cef6a3eace8a1a5b70eb0c8"} Oct 31 19:36:16.814629 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:16.814572 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" podStartSLOduration=1.820794096 podStartE2EDuration="2.814558918s" totalImagesPullingTime="993.764822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:16.813491189 +0000 UTC m=+1148.053714342" watchObservedRunningTime="2026-10-31 19:36:16.814558918 +0000 UTC m=+1148.054782091" Oct 31 19:36:16.842319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:16.842288 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:22.454135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:22.454094 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:36:22.455623 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:22.455587 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:24.843122 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:24.843081 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:24.855569 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:24.855541 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:25.782560 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:25.782525 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" Oct 31 19:36:27.339638 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:36:27.339593 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:36:43.091057 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.091023 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w"] Oct 31 19:36:43.095631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.095612 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:43.116308 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.116275 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tpb\" (UniqueName: \"kubernetes.io/projected/156b35f9-2737-41b0-aab1-37d0887aa557-kube-api-access-t7tpb\") pod \"ai-gateway-controller-5f85cd9cfc-lr89w\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:43.116473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.116438 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-lr89w\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:43.147860 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.147358 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 19:36:43.150601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.150366 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w"] Oct 31 19:36:43.150601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.150505 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 19:36:43.192575 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.192547 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9zrdb\"" Oct 31 19:36:43.217162 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.217129 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-lr89w\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:43.217311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.217192 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tpb\" (UniqueName: \"kubernetes.io/projected/156b35f9-2737-41b0-aab1-37d0887aa557-kube-api-access-t7tpb\") pod \"ai-gateway-controller-5f85cd9cfc-lr89w\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:43.217311 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:36:43.217272 6481 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found Oct 31 19:36:43.217382 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:36:43.217351 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs podName:156b35f9-2737-41b0-aab1-37d0887aa557 nodeName:}" failed. No retries permitted until 2026-10-31 19:36:43.717336901 +0000 UTC m=+1174.957560008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs") pod "ai-gateway-controller-5f85cd9cfc-lr89w" (UID: "156b35f9-2737-41b0-aab1-37d0887aa557") : secret "ai-gateway-controller-metrics-tls" not found Oct 31 19:36:43.270529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.270497 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tpb\" (UniqueName: \"kubernetes.io/projected/156b35f9-2737-41b0-aab1-37d0887aa557-kube-api-access-t7tpb\") pod \"ai-gateway-controller-5f85cd9cfc-lr89w\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:43.721896 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.721856 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-lr89w\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:43.724349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:43.724322 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-lr89w\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:44.008581 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:44.008476 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:44.372810 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:44.372728 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w"] Oct 31 19:36:44.375773 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:36:44.375741 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156b35f9_2737_41b0_aab1_37d0887aa557.slice/crio-b7ede0871b919f68d068c7ae3d8243c68fa7c6f1ee4143d77582c79e4bf9df7d WatchSource:0}: Error finding container b7ede0871b919f68d068c7ae3d8243c68fa7c6f1ee4143d77582c79e4bf9df7d: Status 404 returned error can't find the container with id b7ede0871b919f68d068c7ae3d8243c68fa7c6f1ee4143d77582c79e4bf9df7d Oct 31 19:36:44.912674 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:44.912635 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" event={"ID":"156b35f9-2737-41b0-aab1-37d0887aa557","Type":"ContainerStarted","Data":"b7ede0871b919f68d068c7ae3d8243c68fa7c6f1ee4143d77582c79e4bf9df7d"} Oct 31 19:36:45.653733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:45.653696 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bb479dcdb-n84h4"] Oct 31 19:36:45.654288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:45.654252 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-bb479dcdb-n84h4" podUID="d39947ab-1440-45f9-943d-5c33cd271148" containerName="authorino" containerID="cri-o://8be3148834d006a55d39b4ea873acd88a1e71828a60efded36f2e100128a9fc5" gracePeriod=30 Oct 31 19:36:45.921405 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:45.921231 6481 generic.go:356] "Generic (PLEG): container finished" podID="d39947ab-1440-45f9-943d-5c33cd271148" containerID="8be3148834d006a55d39b4ea873acd88a1e71828a60efded36f2e100128a9fc5" exitCode=0 Oct 31 19:36:45.921405 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:45.921390 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bb479dcdb-n84h4" event={"ID":"d39947ab-1440-45f9-943d-5c33cd271148","Type":"ContainerDied","Data":"8be3148834d006a55d39b4ea873acd88a1e71828a60efded36f2e100128a9fc5"} Oct 31 19:36:45.921611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:45.921431 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bb479dcdb-n84h4" event={"ID":"d39947ab-1440-45f9-943d-5c33cd271148","Type":"ContainerDied","Data":"8b8a5a29e8558aae93cb57cabec8f3fae15634c138de22d4525cb6892c216a88"} Oct 31 19:36:45.921611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:45.921448 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8a5a29e8558aae93cb57cabec8f3fae15634c138de22d4525cb6892c216a88" Oct 31 19:36:45.924683 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:45.924185 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:36:46.044549 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.044516 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39947ab-1440-45f9-943d-5c33cd271148-kube-api-access-n4lzz\" (UniqueName: \"kubernetes.io/projected/d39947ab-1440-45f9-943d-5c33cd271148-kube-api-access-n4lzz\") pod \"d39947ab-1440-45f9-943d-5c33cd271148\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " Oct 31 19:36:46.044721 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.044606 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d39947ab-1440-45f9-943d-5c33cd271148-tls-cert\" (UniqueName: \"kubernetes.io/secret/d39947ab-1440-45f9-943d-5c33cd271148-tls-cert\") pod \"d39947ab-1440-45f9-943d-5c33cd271148\" (UID: \"d39947ab-1440-45f9-943d-5c33cd271148\") " Oct 31 19:36:46.047203 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.047148 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39947ab-1440-45f9-943d-5c33cd271148-kube-api-access-n4lzz" pod "d39947ab-1440-45f9-943d-5c33cd271148" (UID: "d39947ab-1440-45f9-943d-5c33cd271148"). InnerVolumeSpecName "kube-api-access-n4lzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:36:46.058158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.058130 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d39947ab-1440-45f9-943d-5c33cd271148-tls-cert" pod "d39947ab-1440-45f9-943d-5c33cd271148" (UID: "d39947ab-1440-45f9-943d-5c33cd271148"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:36:46.146524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.146437 6481 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d39947ab-1440-45f9-943d-5c33cd271148-tls-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:36:46.146524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.146495 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4lzz\" (UniqueName: \"kubernetes.io/projected/d39947ab-1440-45f9-943d-5c33cd271148-kube-api-access-n4lzz\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:36:46.927134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.927040 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" event={"ID":"156b35f9-2737-41b0-aab1-37d0887aa557","Type":"ContainerStarted","Data":"e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc"} Oct 31 19:36:46.927134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.927055 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bb479dcdb-n84h4" Oct 31 19:36:46.927562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:46.927298 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:36:47.034639 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:47.034588 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" podStartSLOduration=1.7501569460000002 podStartE2EDuration="4.034575169s" totalImagesPullingTime="2.284418223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:47.033219869 +0000 UTC m=+1178.273442999" watchObservedRunningTime="2026-10-31 19:36:47.034575169 +0000 UTC m=+1178.274798298" Oct 31 19:36:47.163641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:47.161548 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bb479dcdb-n84h4"] Oct 31 19:36:47.164699 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:47.164671 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bb479dcdb-n84h4"] Oct 31 19:36:48.878002 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:48.877919 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39947ab-1440-45f9-943d-5c33cd271148" path="/var/lib/kubelet/pods/d39947ab-1440-45f9-943d-5c33cd271148/volumes" Oct 31 19:36:50.887630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.887595 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6"] Oct 31 19:36:50.888059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.887988 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d39947ab-1440-45f9-943d-5c33cd271148" containerName="authorino" Oct 31 19:36:50.888059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.888013 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39947ab-1440-45f9-943d-5c33cd271148" containerName="authorino" Oct 31 19:36:50.888144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.888073 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39947ab-1440-45f9-943d-5c33cd271148" containerName="authorino" Oct 31 19:36:50.892334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.892312 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:50.946192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.946159 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6"] Oct 31 19:36:50.989645 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.989611 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 19:36:50.989896 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:50.989758 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vvdb8\"" Oct 31 19:36:51.000466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.000431 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9095efd-a3e2-4a0d-b9cd-1a380eef1d39-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-5jvp6\" (UID: \"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:51.000635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.000488 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7dt\" (UniqueName: \"kubernetes.io/projected/f9095efd-a3e2-4a0d-b9cd-1a380eef1d39-kube-api-access-8t7dt\") pod \"payload-pre-processing-7b98bb678c-5jvp6\" (UID: \"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:51.054894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.054856 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ff9cw"] Oct 31 19:36:51.059299 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.059275 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.102060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.102018 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7dt\" (UniqueName: \"kubernetes.io/projected/f9095efd-a3e2-4a0d-b9cd-1a380eef1d39-kube-api-access-8t7dt\") pod \"payload-pre-processing-7b98bb678c-5jvp6\" (UID: \"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:51.102230 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.102167 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9095efd-a3e2-4a0d-b9cd-1a380eef1d39-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-5jvp6\" (UID: \"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:51.102847 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.102827 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9095efd-a3e2-4a0d-b9cd-1a380eef1d39-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-5jvp6\" (UID: \"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:51.112237 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.112203 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ff9cw"] Oct 31 19:36:51.159544 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.159466 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7dt\" (UniqueName: \"kubernetes.io/projected/f9095efd-a3e2-4a0d-b9cd-1a380eef1d39-kube-api-access-8t7dt\") pod \"payload-pre-processing-7b98bb678c-5jvp6\" (UID: \"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:51.203367 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.203333 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522c641-6153-4239-90e0-c7a55a55bafb-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ff9cw\" (UID: \"3522c641-6153-4239-90e0-c7a55a55bafb\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.203545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.203400 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfsq\" (UniqueName: \"kubernetes.io/projected/3522c641-6153-4239-90e0-c7a55a55bafb-kube-api-access-bpfsq\") pod \"payload-processing-55fbbbf9dc-ff9cw\" (UID: \"3522c641-6153-4239-90e0-c7a55a55bafb\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.205141 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.205116 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:51.304607 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.304573 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfsq\" (UniqueName: \"kubernetes.io/projected/3522c641-6153-4239-90e0-c7a55a55bafb-kube-api-access-bpfsq\") pod \"payload-processing-55fbbbf9dc-ff9cw\" (UID: \"3522c641-6153-4239-90e0-c7a55a55bafb\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.304779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.304666 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522c641-6153-4239-90e0-c7a55a55bafb-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ff9cw\" (UID: \"3522c641-6153-4239-90e0-c7a55a55bafb\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.305350 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.305328 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522c641-6153-4239-90e0-c7a55a55bafb-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ff9cw\" (UID: \"3522c641-6153-4239-90e0-c7a55a55bafb\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.358712 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.358685 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfsq\" (UniqueName: \"kubernetes.io/projected/3522c641-6153-4239-90e0-c7a55a55bafb-kube-api-access-bpfsq\") pod \"payload-processing-55fbbbf9dc-ff9cw\" (UID: \"3522c641-6153-4239-90e0-c7a55a55bafb\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.378259 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.378234 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:51.582364 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.582335 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6"] Oct 31 19:36:51.584608 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:36:51.584578 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9095efd_a3e2_4a0d_b9cd_1a380eef1d39.slice/crio-254bc74b41d6575e3af587d066ad41d360512ff97965a83ac5737f31dd280602 WatchSource:0}: Error finding container 254bc74b41d6575e3af587d066ad41d360512ff97965a83ac5737f31dd280602: Status 404 returned error can't find the container with id 254bc74b41d6575e3af587d066ad41d360512ff97965a83ac5737f31dd280602 Oct 31 19:36:51.740444 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.740315 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ff9cw"] Oct 31 19:36:51.742943 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:36:51.742915 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3522c641_6153_4239_90e0_c7a55a55bafb.slice/crio-e9ec8840402b84f3d87c958075053426aeb07c6818b282b301f0d78ace466309 WatchSource:0}: Error finding container e9ec8840402b84f3d87c958075053426aeb07c6818b282b301f0d78ace466309: Status 404 returned error can't find the container with id e9ec8840402b84f3d87c958075053426aeb07c6818b282b301f0d78ace466309 Oct 31 19:36:51.951155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.951116 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" event={"ID":"3522c641-6153-4239-90e0-c7a55a55bafb","Type":"ContainerStarted","Data":"e9ec8840402b84f3d87c958075053426aeb07c6818b282b301f0d78ace466309"} Oct 31 19:36:51.952337 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:51.952311 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" event={"ID":"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39","Type":"ContainerStarted","Data":"254bc74b41d6575e3af587d066ad41d360512ff97965a83ac5737f31dd280602"} Oct 31 19:36:54.971062 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:54.971022 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" event={"ID":"3522c641-6153-4239-90e0-c7a55a55bafb","Type":"ContainerStarted","Data":"e48758f62384b1392df11886c8ed4b862597f1370c95985139307e87875fa694"} Oct 31 19:36:54.971534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:54.971161 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:36:54.972701 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:54.972676 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" event={"ID":"f9095efd-a3e2-4a0d-b9cd-1a380eef1d39","Type":"ContainerStarted","Data":"5394f1edef0a6fae752e41d8b71e7a8b1ff961ed3c1c263b8c6abdf1aba4ddcf"} Oct 31 19:36:54.972796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:54.972746 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:36:55.078184 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:55.078093 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" podStartSLOduration=0.948206106 podStartE2EDuration="4.078079566s" totalImagesPullingTime="3.12987346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:55.077341182 +0000 UTC m=+1186.317564311" watchObservedRunningTime="2026-10-31 19:36:55.078079566 +0000 UTC m=+1186.318302730" Oct 31 19:36:55.183692 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:55.183634 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" podStartSLOduration=1.989157864 podStartE2EDuration="5.183621468s" totalImagesPullingTime="3.194463604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:55.183058814 +0000 UTC m=+1186.423281945" watchObservedRunningTime="2026-10-31 19:36:55.183621468 +0000 UTC m=+1186.423844597" Oct 31 19:36:57.323724 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:36:57.323683 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:36:57.933930 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:36:57.933899 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:37:05.979877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:05.979846 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ff9cw" Oct 31 19:37:05.980357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:05.979914 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5jvp6" Oct 31 19:37:13.183045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.182989 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w"] Oct 31 19:37:13.183477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.183240 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" podUID="156b35f9-2737-41b0-aab1-37d0887aa557" containerName="manager" containerID="cri-o://e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc" gracePeriod=30 Oct 31 19:37:13.440911 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.440850 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:37:13.617427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.617395 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs\" (UniqueName: \"kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs\") pod \"156b35f9-2737-41b0-aab1-37d0887aa557\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " Oct 31 19:37:13.617600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.617474 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156b35f9-2737-41b0-aab1-37d0887aa557-kube-api-access-t7tpb\" (UniqueName: \"kubernetes.io/projected/156b35f9-2737-41b0-aab1-37d0887aa557-kube-api-access-t7tpb\") pod \"156b35f9-2737-41b0-aab1-37d0887aa557\" (UID: \"156b35f9-2737-41b0-aab1-37d0887aa557\") " Oct 31 19:37:13.619540 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.619503 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs" pod "156b35f9-2737-41b0-aab1-37d0887aa557" (UID: "156b35f9-2737-41b0-aab1-37d0887aa557"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:37:13.619656 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.619560 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156b35f9-2737-41b0-aab1-37d0887aa557-kube-api-access-t7tpb" pod "156b35f9-2737-41b0-aab1-37d0887aa557" (UID: "156b35f9-2737-41b0-aab1-37d0887aa557"). InnerVolumeSpecName "kube-api-access-t7tpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:37:13.718206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.718180 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7tpb\" (UniqueName: \"kubernetes.io/projected/156b35f9-2737-41b0-aab1-37d0887aa557-kube-api-access-t7tpb\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:37:13.718206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:13.718206 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156b35f9-2737-41b0-aab1-37d0887aa557-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:37:14.056856 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.056774 6481 generic.go:356] "Generic (PLEG): container finished" podID="156b35f9-2737-41b0-aab1-37d0887aa557" containerID="e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc" exitCode=0 Oct 31 19:37:14.056856 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.056818 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" Oct 31 19:37:14.057191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.056818 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" event={"ID":"156b35f9-2737-41b0-aab1-37d0887aa557","Type":"ContainerDied","Data":"e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc"} Oct 31 19:37:14.057191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.056950 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w" event={"ID":"156b35f9-2737-41b0-aab1-37d0887aa557","Type":"ContainerDied","Data":"b7ede0871b919f68d068c7ae3d8243c68fa7c6f1ee4143d77582c79e4bf9df7d"} Oct 31 19:37:14.057191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.056978 6481 scope.go:122] "RemoveContainer" containerID="e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc" Oct 31 19:37:14.066515 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.066500 6481 scope.go:122] "RemoveContainer" containerID="e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc" Oct 31 19:37:14.066802 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:37:14.066783 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc\": container with ID starting with e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc not found: ID does not exist" containerID="e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc" Oct 31 19:37:14.066858 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.066810 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc"} err="failed to get container status \"e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc\": rpc error: code = NotFound desc = could not find container \"e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc\": container with ID starting with e206fbd5065f6119fbd0fc71fbbf2bad3bdca887ad5f11a06b512922690537bc not found: ID does not exist" Oct 31 19:37:14.208641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.208608 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w"] Oct 31 19:37:14.254633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.254602 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-lr89w"] Oct 31 19:37:14.878160 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:14.878130 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156b35f9-2737-41b0-aab1-37d0887aa557" path="/var/lib/kubelet/pods/156b35f9-2737-41b0-aab1-37d0887aa557/volumes" Oct 31 19:37:27.323337 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:37:27.323301 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:37:43.496714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.496676 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d"] Oct 31 19:37:43.497312 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.497292 6481 cpu_manager.go:414] "containerMap: removing container" podUID="156b35f9-2737-41b0-aab1-37d0887aa557" containerName="manager" Oct 31 19:37:43.497399 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.497314 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156b35f9-2737-41b0-aab1-37d0887aa557" containerName="manager" Oct 31 19:37:43.497476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.497463 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="156b35f9-2737-41b0-aab1-37d0887aa557" containerName="manager" Oct 31 19:37:43.500634 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.500611 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.565199 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.565170 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5a4438-openshift-default-dockercfg-7kvr6\"" Oct 31 19:37:43.591471 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591443 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.591605 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591555 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.591605 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591600 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.591720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591625 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.591787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591725 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.591838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591804 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.591949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591889 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66z7\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-kube-api-access-f66z7\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.592098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.591973 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.592098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.592029 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.597387 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.597364 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d"] Oct 31 19:37:43.692891 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.692862 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.692931 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.692981 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693031 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693058 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693098 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693133 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66z7\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-kube-api-access-f66z7\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693163 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693194 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693348 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693441 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693538 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693614 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.693902 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.693881 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.695496 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.695474 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.695586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.695496 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.746597 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.746565 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66z7\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-kube-api-access-f66z7\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.746832 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.746787 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:43.813323 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:43.813300 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:44.230109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.230085 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d"] Oct 31 19:37:44.233186 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:37:44.233156 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d20cc7_1819_4c64_8c38_c4ad81f1b469.slice/crio-a14b29f762839e94e88e30b7555b80a3e65dd7ba10fa6caf17baa067d9b6befb WatchSource:0}: Error finding container a14b29f762839e94e88e30b7555b80a3e65dd7ba10fa6caf17baa067d9b6befb: Status 404 returned error can't find the container with id a14b29f762839e94e88e30b7555b80a3e65dd7ba10fa6caf17baa067d9b6befb Oct 31 19:37:44.235318 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.235285 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:44.235434 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.235349 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:44.235434 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.235376 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:44.389409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.389376 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k"] Oct 31 19:37:44.392862 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.392843 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.441453 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.441424 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-093b97-openshift-default-dockercfg-mndxw\"" Oct 31 19:37:44.450495 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.450464 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k"] Oct 31 19:37:44.502299 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502212 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502299 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502275 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502354 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrm5\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-kube-api-access-fdrm5\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502393 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502425 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502511 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502547 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026ac1-6520-413a-aa94-86e7f85f5578-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502575 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.502806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.502616 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026ac1-6520-413a-aa94-86e7f85f5578-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.603848 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.603819 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.603855 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrm5\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-kube-api-access-fdrm5\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.603879 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.603902 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.603957 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.603985 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026ac1-6520-413a-aa94-86e7f85f5578-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604322 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604074 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604322 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604112 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026ac1-6520-413a-aa94-86e7f85f5578-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604322 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604182 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604469 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604321 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604637 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604654 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604700 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.604888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.604811 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026ac1-6520-413a-aa94-86e7f85f5578-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.606256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.606235 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.606820 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.606800 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026ac1-6520-413a-aa94-86e7f85f5578-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.660602 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.660570 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.660736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.660677 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrm5\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-kube-api-access-fdrm5\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.689191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.689163 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8"] Oct 31 19:37:44.693931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.693909 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.708558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.708533 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:44.758342 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.758311 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-967dee-openshift-default-dockercfg-btwn6\"" Oct 31 19:37:44.759333 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.759308 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8"] Oct 31 19:37:44.807794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.807722 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.807922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.807831 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b383e843-8685-4714-b21a-ded1019edf5b-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.807922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.807873 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.808052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.807949 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.808052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.807975 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b383e843-8685-4714-b21a-ded1019edf5b-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.808165 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.808097 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.808165 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.808118 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.808165 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.808148 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.808316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.808166 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xvj\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-kube-api-access-l6xvj\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.909675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.909644 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.909675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.909677 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xvj\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-kube-api-access-l6xvj\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.909917 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.909716 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.909917 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.909751 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b383e843-8685-4714-b21a-ded1019edf5b-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.909917 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.909765 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.909917 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.909905 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.909952 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b383e843-8685-4714-b21a-ded1019edf5b-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.910033 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.910065 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.910117 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.910197 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.910395 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.910421 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.910659 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.910640 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b383e843-8685-4714-b21a-ded1019edf5b-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.912193 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.912162 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.912446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.912429 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b383e843-8685-4714-b21a-ded1019edf5b-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.962832 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.962811 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:44.962943 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:44.962865 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xvj\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-kube-api-access-l6xvj\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-k56f8\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:45.061049 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.060971 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:45.123956 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.123923 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k"] Oct 31 19:37:45.127199 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:37:45.127169 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99026ac1_6520_413a_aa94_86e7f85f5578.slice/crio-c81b3a892ba0ea31ac3319f38fb7821e70a4eb9446da881338d323326d6eec42 WatchSource:0}: Error finding container c81b3a892ba0ea31ac3319f38fb7821e70a4eb9446da881338d323326d6eec42: Status 404 returned error can't find the container with id c81b3a892ba0ea31ac3319f38fb7821e70a4eb9446da881338d323326d6eec42 Oct 31 19:37:45.131800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.131387 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:45.131800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.131453 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:45.131800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.131482 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:45.201574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.201543 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" event={"ID":"67d20cc7-1819-4c64-8c38-c4ad81f1b469","Type":"ContainerStarted","Data":"5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e"} Oct 31 19:37:45.201753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.201596 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" event={"ID":"67d20cc7-1819-4c64-8c38-c4ad81f1b469","Type":"ContainerStarted","Data":"a14b29f762839e94e88e30b7555b80a3e65dd7ba10fa6caf17baa067d9b6befb"} Oct 31 19:37:45.203026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.202985 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" event={"ID":"99026ac1-6520-413a-aa94-86e7f85f5578","Type":"ContainerStarted","Data":"c81b3a892ba0ea31ac3319f38fb7821e70a4eb9446da881338d323326d6eec42"} Oct 31 19:37:45.292327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.292295 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb"] Oct 31 19:37:45.295929 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.295911 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.316027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.315921 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podStartSLOduration=2.315906901 podStartE2EDuration="2.315906901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:45.315057375 +0000 UTC m=+1236.555280504" watchObservedRunningTime="2026-10-31 19:37:45.315906901 +0000 UTC m=+1236.556130029" Oct 31 19:37:45.344738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.344699 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7cfafd-openshift-default-dockercfg-96gwp\"" Oct 31 19:37:45.351618 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.351248 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb"] Oct 31 19:37:45.415405 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415370 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.415604 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415441 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ss5m\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-kube-api-access-6ss5m\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.415604 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415556 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.415715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415620 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.415715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415650 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.415791 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415768 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.415828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415807 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.415923 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.415895 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.416029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.416009 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.449971 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.449948 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8"] Oct 31 19:37:45.452714 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:37:45.452686 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb383e843_8685_4714_b21a_ded1019edf5b.slice/crio-72f76978964638a8e095bdb5b2b7f0a473ded5c78a7fcd00657e5a39e30a8498 WatchSource:0}: Error finding container 72f76978964638a8e095bdb5b2b7f0a473ded5c78a7fcd00657e5a39e30a8498: Status 404 returned error can't find the container with id 72f76978964638a8e095bdb5b2b7f0a473ded5c78a7fcd00657e5a39e30a8498 Oct 31 19:37:45.455022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.454973 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:45.455101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.455046 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:45.455101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.455079 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:45.518016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.517740 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.517804 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.517843 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.517898 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.517947 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.517988 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518731 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.518067 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ss5m\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-kube-api-access-6ss5m\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518731 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.518111 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.518731 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.518178 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.520088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.519260 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.520088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.519565 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.520088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.519825 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.520420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.520149 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.520420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.520372 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.527221 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.527195 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.531024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.530981 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.570910 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.570848 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.576041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.576018 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ss5m\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-kube-api-access-6ss5m\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.614892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.614863 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:45.813748 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.813715 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:45.819380 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:45.819338 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:46.011851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.011827 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb"] Oct 31 19:37:46.014458 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:37:46.014431 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d75b9d6_fe3f_4b0e_afde_d3995902d013.slice/crio-8f7b26119d8464130961c5a4870a8e1ed3c1937d78dd84064c6cb4659986f3f3 WatchSource:0}: Error finding container 8f7b26119d8464130961c5a4870a8e1ed3c1937d78dd84064c6cb4659986f3f3: Status 404 returned error can't find the container with id 8f7b26119d8464130961c5a4870a8e1ed3c1937d78dd84064c6cb4659986f3f3 Oct 31 19:37:46.016923 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.016882 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:46.017043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.016966 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:46.017112 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.017039 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:37:46.209535 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.209498 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" event={"ID":"8d75b9d6-fe3f-4b0e-afde-d3995902d013","Type":"ContainerStarted","Data":"57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138"} Oct 31 19:37:46.209716 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.209549 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" event={"ID":"8d75b9d6-fe3f-4b0e-afde-d3995902d013","Type":"ContainerStarted","Data":"8f7b26119d8464130961c5a4870a8e1ed3c1937d78dd84064c6cb4659986f3f3"} Oct 31 19:37:46.211239 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.211214 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" event={"ID":"99026ac1-6520-413a-aa94-86e7f85f5578","Type":"ContainerStarted","Data":"ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42"} Oct 31 19:37:46.212796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.212774 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" event={"ID":"b383e843-8685-4714-b21a-ded1019edf5b","Type":"ContainerStarted","Data":"7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213"} Oct 31 19:37:46.212900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.212810 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" event={"ID":"b383e843-8685-4714-b21a-ded1019edf5b","Type":"ContainerStarted","Data":"72f76978964638a8e095bdb5b2b7f0a473ded5c78a7fcd00657e5a39e30a8498"} Oct 31 19:37:46.213131 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.213107 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:46.214093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.214074 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:37:46.319379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.319264 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podStartSLOduration=1.319249759 podStartE2EDuration="1.319249759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:46.318659556 +0000 UTC m=+1237.558882688" watchObservedRunningTime="2026-10-31 19:37:46.319249759 +0000 UTC m=+1237.559472966" Oct 31 19:37:46.615381 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.615309 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:46.673456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.673402 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" podStartSLOduration=2.673386067 podStartE2EDuration="2.673386067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:46.551687184 +0000 UTC m=+1237.791910314" watchObservedRunningTime="2026-10-31 19:37:46.673386067 +0000 UTC m=+1237.913609199" Oct 31 19:37:46.673828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.673805 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" podStartSLOduration=2.673797096 podStartE2EDuration="2.673797096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:46.672308937 +0000 UTC m=+1237.912532067" watchObservedRunningTime="2026-10-31 19:37:46.673797096 +0000 UTC m=+1237.914020224" Oct 31 19:37:46.708779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.708739 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:46.715330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:46.715299 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:47.061743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:47.061708 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:47.067405 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:47.067380 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:47.217981 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:47.217688 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:47.217981 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:47.217739 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:47.218429 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:47.218398 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:37:47.218543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:47.218520 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:37:47.620307 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:47.620275 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:48.222920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:48.222871 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:48.223305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:48.223287 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:37:57.323755 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:37:57.323717 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:37:58.910740 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:58.910705 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj"] Oct 31 19:37:58.914157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:58.914138 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:58.965440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:58.965393 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad37122-9855-4567-aba6-c00abc4eefb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:58.965788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:58.965763 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5jn\" (UniqueName: \"kubernetes.io/projected/4ad37122-9855-4567-aba6-c00abc4eefb8-kube-api-access-9n5jn\") pod \"payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:58.966545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:58.966522 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj"] Oct 31 19:37:59.013031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.010259 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b5155c-dockercfg-pnxnm\"" Oct 31 19:37:59.013031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.010394 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b5155c\"" Oct 31 19:37:59.066580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.066538 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad37122-9855-4567-aba6-c00abc4eefb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:59.066738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.066633 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5jn\" (UniqueName: \"kubernetes.io/projected/4ad37122-9855-4567-aba6-c00abc4eefb8-kube-api-access-9n5jn\") pod \"payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:59.067330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.067311 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad37122-9855-4567-aba6-c00abc4eefb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:59.119469 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.119437 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5jn\" (UniqueName: \"kubernetes.io/projected/4ad37122-9855-4567-aba6-c00abc4eefb8-kube-api-access-9n5jn\") pod \"payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:59.226940 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.226912 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:37:59.603270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:37:59.603243 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj"] Oct 31 19:37:59.605575 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:37:59.605548 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad37122_9855_4567_aba6_c00abc4eefb8.slice/crio-24a26e27e2b462c9d9b49cacbd05dd26a630a47632ab1f6a400fbf539212bf84 WatchSource:0}: Error finding container 24a26e27e2b462c9d9b49cacbd05dd26a630a47632ab1f6a400fbf539212bf84: Status 404 returned error can't find the container with id 24a26e27e2b462c9d9b49cacbd05dd26a630a47632ab1f6a400fbf539212bf84 Oct 31 19:38:00.068782 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.068749 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l"] Oct 31 19:38:00.072490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.072472 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.125708 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.125666 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l"] Oct 31 19:38:00.169373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.169341 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 19:38:00.169373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.169341 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-662be1-serving-cert\"" Oct 31 19:38:00.169579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.169383 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xd7wl\"" Oct 31 19:38:00.179790 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.179765 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.179962 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.179795 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba519c-3954-4e63-ac58-384d264c1494-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.180067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.179989 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-metrics-certs\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.180113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.180102 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59v6h\" (UniqueName: \"kubernetes.io/projected/7eba519c-3954-4e63-ac58-384d264c1494-kube-api-access-59v6h\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.281728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.281688 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-metrics-certs\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.281904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.281743 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59v6h\" (UniqueName: \"kubernetes.io/projected/7eba519c-3954-4e63-ac58-384d264c1494-kube-api-access-59v6h\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.281904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.281816 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.281904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.281839 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba519c-3954-4e63-ac58-384d264c1494-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.282120 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:00.281964 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-serving-cert: secret "maas-api-e2e-worker-w3-662be1-serving-cert" not found Oct 31 19:38:00.282120 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:00.282063 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls podName:7eba519c-3954-4e63-ac58-384d264c1494 nodeName:}" failed. No retries permitted until 2026-10-31 19:38:00.782042437 +0000 UTC m=+1252.022265565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls") pod "maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" (UID: "7eba519c-3954-4e63-ac58-384d264c1494") : secret "maas-api-e2e-worker-w3-662be1-serving-cert" not found Oct 31 19:38:00.282728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.282702 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba519c-3954-4e63-ac58-384d264c1494-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.284302 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.284279 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-metrics-certs\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.288676 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.288651 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" event={"ID":"4ad37122-9855-4567-aba6-c00abc4eefb8","Type":"ContainerStarted","Data":"de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4"} Oct 31 19:38:00.288769 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.288703 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" event={"ID":"4ad37122-9855-4567-aba6-c00abc4eefb8","Type":"ContainerStarted","Data":"24a26e27e2b462c9d9b49cacbd05dd26a630a47632ab1f6a400fbf539212bf84"} Oct 31 19:38:00.288818 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.288802 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:38:00.339580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.339499 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59v6h\" (UniqueName: \"kubernetes.io/projected/7eba519c-3954-4e63-ac58-384d264c1494-kube-api-access-59v6h\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.396571 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.396514 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" podStartSLOduration=2.3965004690000002 podStartE2EDuration="2.396500469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:00.394099223 +0000 UTC m=+1251.634322354" watchObservedRunningTime="2026-10-31 19:38:00.396500469 +0000 UTC m=+1251.636723624" Oct 31 19:38:00.796074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.796040 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.798375 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.798349 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls\") pod \"maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:00.985287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:00.985257 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:01.424636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:01.417959 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l"] Oct 31 19:38:01.439564 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:38:01.438795 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eba519c_3954_4e63_ac58_384d264c1494.slice/crio-9d0d444842127a0c21d1ea751cec853f143daced2109ec42ca3654e8e94dcfd2 WatchSource:0}: Error finding container 9d0d444842127a0c21d1ea751cec853f143daced2109ec42ca3654e8e94dcfd2: Status 404 returned error can't find the container with id 9d0d444842127a0c21d1ea751cec853f143daced2109ec42ca3654e8e94dcfd2 Oct 31 19:38:02.300413 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:02.300375 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" event={"ID":"7eba519c-3954-4e63-ac58-384d264c1494","Type":"ContainerStarted","Data":"9d0d444842127a0c21d1ea751cec853f143daced2109ec42ca3654e8e94dcfd2"} Oct 31 19:38:03.306880 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:03.306834 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" event={"ID":"7eba519c-3954-4e63-ac58-384d264c1494","Type":"ContainerStarted","Data":"76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314"} Oct 31 19:38:03.307324 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:03.306940 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:03.417703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:03.417645 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" podStartSLOduration=2.456328822 podStartE2EDuration="3.417630588s" totalImagesPullingTime="961.301766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:03.416071929 +0000 UTC m=+1254.656295058" watchObservedRunningTime="2026-10-31 19:38:03.417630588 +0000 UTC m=+1254.657853746" Oct 31 19:38:05.528212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.528174 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth"] Oct 31 19:38:05.532056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.532032 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.580438 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.580409 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-5a4438-serving-cert\"" Oct 31 19:38:05.629982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.629943 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth"] Oct 31 19:38:05.673241 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.673206 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-metrics-certs\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.673459 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.673299 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.673459 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.673338 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece716af-28ae-4bb2-a370-e20910206680-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.673459 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.673432 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glmdq\" (UniqueName: \"kubernetes.io/projected/ece716af-28ae-4bb2-a370-e20910206680-kube-api-access-glmdq\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.774889 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.774857 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-metrics-certs\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.775074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.775046 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.775148 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.775102 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece716af-28ae-4bb2-a370-e20910206680-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.775194 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:05.775156 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-serving-cert: secret "maas-api-e2e-worker-w6-5a4438-serving-cert" not found Oct 31 19:38:05.775229 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.775198 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glmdq\" (UniqueName: \"kubernetes.io/projected/ece716af-28ae-4bb2-a370-e20910206680-kube-api-access-glmdq\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.775229 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:05.775214 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls podName:ece716af-28ae-4bb2-a370-e20910206680 nodeName:}" failed. No retries permitted until 2026-10-31 19:38:06.275196768 +0000 UTC m=+1257.515419896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls") pod "maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" (UID: "ece716af-28ae-4bb2-a370-e20910206680") : secret "maas-api-e2e-worker-w6-5a4438-serving-cert" not found Oct 31 19:38:05.775871 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.775843 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece716af-28ae-4bb2-a370-e20910206680-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.777541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.777517 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-metrics-certs\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:05.829332 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:05.829241 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glmdq\" (UniqueName: \"kubernetes.io/projected/ece716af-28ae-4bb2-a370-e20910206680-kube-api-access-glmdq\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:06.281894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:06.281850 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:06.284473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:06.284437 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:06.445733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:06.445698 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:06.812554 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:06.812526 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth"] Oct 31 19:38:06.814621 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:38:06.814582 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece716af_28ae_4bb2_a370_e20910206680.slice/crio-a6005b899db6f278f9c414fdac5bd36da392a0ae368240d72303eb6d2d86b37f WatchSource:0}: Error finding container a6005b899db6f278f9c414fdac5bd36da392a0ae368240d72303eb6d2d86b37f: Status 404 returned error can't find the container with id a6005b899db6f278f9c414fdac5bd36da392a0ae368240d72303eb6d2d86b37f Oct 31 19:38:07.328828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:07.328790 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" event={"ID":"ece716af-28ae-4bb2-a370-e20910206680","Type":"ContainerStarted","Data":"a6005b899db6f278f9c414fdac5bd36da392a0ae368240d72303eb6d2d86b37f"} Oct 31 19:38:08.335073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:08.335035 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" event={"ID":"ece716af-28ae-4bb2-a370-e20910206680","Type":"ContainerStarted","Data":"e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae"} Oct 31 19:38:08.335468 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:08.335096 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:08.443322 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:08.443261 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" podStartSLOduration=2.5647792430000003 podStartE2EDuration="3.443241435s" totalImagesPullingTime="878.462192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:08.442327007 +0000 UTC m=+1259.682550137" watchObservedRunningTime="2026-10-31 19:38:08.443241435 +0000 UTC m=+1259.683464566" Oct 31 19:38:09.316715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:09.316628 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 19:38:11.362580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:11.362553 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 19:38:14.345562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:14.345532 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 19:38:15.748956 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.748921 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7"] Oct 31 19:38:15.755732 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.755705 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.816963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.801176 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff52ec1e-fa64-4694-8104-83ae79287e85-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.816963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.816190 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zcm9\" (UniqueName: \"kubernetes.io/projected/ff52ec1e-fa64-4694-8104-83ae79287e85-kube-api-access-4zcm9\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.816963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.816409 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.816963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.816534 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-metrics-certs\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.816963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.808314 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7"] Oct 31 19:38:15.816963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.808517 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b5155c-serving-cert\"" Oct 31 19:38:15.917864 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.917824 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.918153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.917896 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-metrics-certs\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.918153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.917936 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff52ec1e-fa64-4694-8104-83ae79287e85-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.918153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.917961 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zcm9\" (UniqueName: \"kubernetes.io/projected/ff52ec1e-fa64-4694-8104-83ae79287e85-kube-api-access-4zcm9\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.918153 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:15.918033 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-serving-cert: secret "maas-api-e2e-worker-w1-b5155c-serving-cert" not found Oct 31 19:38:15.918153 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:15.918118 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls podName:ff52ec1e-fa64-4694-8104-83ae79287e85 nodeName:}" failed. No retries permitted until 2026-10-31 19:38:16.418096436 +0000 UTC m=+1267.658319544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls") pod "maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" (UID: "ff52ec1e-fa64-4694-8104-83ae79287e85") : secret "maas-api-e2e-worker-w1-b5155c-serving-cert" not found Oct 31 19:38:15.918722 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.918692 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff52ec1e-fa64-4694-8104-83ae79287e85-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.920649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.920625 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-metrics-certs\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:15.973691 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:15.973654 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zcm9\" (UniqueName: \"kubernetes.io/projected/ff52ec1e-fa64-4694-8104-83ae79287e85-kube-api-access-4zcm9\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:16.425289 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:16.425250 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:16.427705 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:16.427673 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:16.672295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:16.672261 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:17.036645 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:17.036618 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7"] Oct 31 19:38:17.039034 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:38:17.038986 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff52ec1e_fa64_4694_8104_83ae79287e85.slice/crio-83a857c92fc90b5039d9d6edbb0d1178ff0dc810d4b2254c8a3281487760d9a3 WatchSource:0}: Error finding container 83a857c92fc90b5039d9d6edbb0d1178ff0dc810d4b2254c8a3281487760d9a3: Status 404 returned error can't find the container with id 83a857c92fc90b5039d9d6edbb0d1178ff0dc810d4b2254c8a3281487760d9a3 Oct 31 19:38:17.040515 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:17.040495 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:38:17.398391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:17.398305 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" event={"ID":"ff52ec1e-fa64-4694-8104-83ae79287e85","Type":"ContainerStarted","Data":"83a857c92fc90b5039d9d6edbb0d1178ff0dc810d4b2254c8a3281487760d9a3"} Oct 31 19:38:18.430281 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:18.430203 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" event={"ID":"ff52ec1e-fa64-4694-8104-83ae79287e85","Type":"ContainerStarted","Data":"1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a"} Oct 31 19:38:18.430946 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:18.430925 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:25.448130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:25.448101 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 19:38:25.563347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:25.563296 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" podStartSLOduration=9.492062555 podStartE2EDuration="10.56328117s" totalImagesPullingTime="1.071218615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:18.573649268 +0000 UTC m=+1269.813872398" watchObservedRunningTime="2026-10-31 19:38:25.56328117 +0000 UTC m=+1276.803504298" Oct 31 19:38:26.139658 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.139618 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9"] Oct 31 19:38:26.143328 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.143307 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.194674 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.194639 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-570915-serving-cert\"" Oct 31 19:38:26.197369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.197336 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9"] Oct 31 19:38:26.219889 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.219846 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.220132 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.219961 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.220132 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.220059 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c84s\" (UniqueName: \"kubernetes.io/projected/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-kube-api-access-5c84s\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.220273 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.220129 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-metrics-certs\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.321912 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.321877 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-metrics-certs\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.322111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.321964 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.322111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.322027 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.322248 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.322113 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c84s\" (UniqueName: \"kubernetes.io/projected/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-kube-api-access-5c84s\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.322248 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:26.322195 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-serving-cert: secret "maas-api-e2e-worker-w5-570915-serving-cert" not found Oct 31 19:38:26.322346 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:26.322294 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls podName:4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3 nodeName:}" failed. No retries permitted until 2026-10-31 19:38:26.822271114 +0000 UTC m=+1278.062494242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls") pod "maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" (UID: "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3") : secret "maas-api-e2e-worker-w5-570915-serving-cert" not found Oct 31 19:38:26.322738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.322720 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.324296 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.324277 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-metrics-certs\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.378031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.377986 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c84s\" (UniqueName: \"kubernetes.io/projected/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-kube-api-access-5c84s\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.828404 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.828368 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:26.830747 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:26.830722 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls\") pod \"maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:27.058019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.057916 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:27.325793 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:27.325759 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:38:27.423430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.423405 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9"] Oct 31 19:38:27.425722 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:38:27.425689 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb91e5e_1bf7_4449_9c55_c1fb2a6211e3.slice/crio-306337b133c50705afc6f9008322f3a4eeee850fc0905f5a67bee52e0aac4b5a WatchSource:0}: Error finding container 306337b133c50705afc6f9008322f3a4eeee850fc0905f5a67bee52e0aac4b5a: Status 404 returned error can't find the container with id 306337b133c50705afc6f9008322f3a4eeee850fc0905f5a67bee52e0aac4b5a Oct 31 19:38:27.482065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.482027 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" event={"ID":"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3","Type":"ContainerStarted","Data":"306337b133c50705afc6f9008322f3a4eeee850fc0905f5a67bee52e0aac4b5a"} Oct 31 19:38:27.483596 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.483567 6481 generic.go:356] "Generic (PLEG): container finished" podID="99026ac1-6520-413a-aa94-86e7f85f5578" containerID="ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42" exitCode=0 Oct 31 19:38:27.483730 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.483637 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" event={"ID":"99026ac1-6520-413a-aa94-86e7f85f5578","Type":"ContainerDied","Data":"ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42"} Oct 31 19:38:27.484152 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.484133 6481 scope.go:122] "RemoveContainer" containerID="ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42" Oct 31 19:38:27.485252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.485232 6481 generic.go:356] "Generic (PLEG): container finished" podID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerID="5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e" exitCode=0 Oct 31 19:38:27.485347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.485284 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" event={"ID":"67d20cc7-1819-4c64-8c38-c4ad81f1b469","Type":"ContainerDied","Data":"5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e"} Oct 31 19:38:27.485599 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.485586 6481 scope.go:122] "RemoveContainer" containerID="5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e" Oct 31 19:38:27.485641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.485605 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:27.485687 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.485673 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:27.485723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.485707 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:27.486735 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.486710 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:27.486802 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.486772 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:27.486841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:27.486806 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:28.523921 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.523878 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" event={"ID":"99026ac1-6520-413a-aa94-86e7f85f5578","Type":"ContainerStarted","Data":"ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a"} Oct 31 19:38:28.525890 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.525863 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" event={"ID":"67d20cc7-1819-4c64-8c38-c4ad81f1b469","Type":"ContainerStarted","Data":"a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388"} Oct 31 19:38:28.527813 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.527786 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" event={"ID":"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3","Type":"ContainerStarted","Data":"c131fde193b4e0a85afb4e45b3ee8ac1ba40ffe0e671ea45de9d80c5c57783e6"} Oct 31 19:38:28.528255 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.528238 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:28.714017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.712361 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:38:28.716226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.716112 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:38:28.716226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.716182 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 19:38:28.764733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.764681 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" podStartSLOduration=1.860932135 podStartE2EDuration="2.764665856s" totalImagesPullingTime="903.733721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:28.762925094 +0000 UTC m=+1280.003148225" watchObservedRunningTime="2026-10-31 19:38:28.764665856 +0000 UTC m=+1280.004888984" Oct 31 19:38:28.818058 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.818033 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:38:28.819441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.819421 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:38:28.819623 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:28.819602 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 19:38:29.710464 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:29.709370 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:38:29.710464 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:29.709438 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 19:38:29.815914 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:29.815815 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:38:29.815914 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:29.815896 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 19:38:30.715425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:30.715380 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:38:30.715922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:30.715442 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 19:38:30.814843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:30.814808 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:38:30.815152 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:30.815121 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 19:38:31.216232 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:31.216199 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:38:31.714379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:31.714346 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:38:31.714635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:31.714610 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 19:38:31.823022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:31.819525 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:38:31.823022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:31.819597 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 19:38:32.218364 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:32.218319 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:38:32.715490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:32.715455 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:38:32.715770 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:32.715740 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 19:38:32.817021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:32.816694 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:38:32.817021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:32.816753 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 19:38:33.781904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:33.781867 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:38:33.785672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:33.785643 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 19:38:33.821210 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:33.821179 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:38:33.822169 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:33.822153 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:38:35.546098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:35.546069 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 19:38:41.336366 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.336332 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j"] Oct 31 19:38:41.339338 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.339320 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.394587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.394353 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-967dee-serving-cert\"" Oct 31 19:38:41.442420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.442384 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j"] Oct 31 19:38:41.458658 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.458624 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82621e2b-ff5a-45b2-b468-5565a05f7f0a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.458840 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.458671 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqtr\" (UniqueName: \"kubernetes.io/projected/82621e2b-ff5a-45b2-b468-5565a05f7f0a-kube-api-access-prqtr\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.458840 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.458779 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.458913 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.458839 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-metrics-certs\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.560482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.560445 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.560676 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.560503 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-metrics-certs\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.560676 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.560572 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82621e2b-ff5a-45b2-b468-5565a05f7f0a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.560676 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.560602 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqtr\" (UniqueName: \"kubernetes.io/projected/82621e2b-ff5a-45b2-b468-5565a05f7f0a-kube-api-access-prqtr\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.560676 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:41.560603 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-serving-cert: secret "maas-api-e2e-worker-w2-967dee-serving-cert" not found Oct 31 19:38:41.560676 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:41.560670 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls podName:82621e2b-ff5a-45b2-b468-5565a05f7f0a nodeName:}" failed. No retries permitted until 2026-10-31 19:38:42.060653837 +0000 UTC m=+1293.300876962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls") pod "maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" (UID: "82621e2b-ff5a-45b2-b468-5565a05f7f0a") : secret "maas-api-e2e-worker-w2-967dee-serving-cert" not found Oct 31 19:38:41.561310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.561287 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82621e2b-ff5a-45b2-b468-5565a05f7f0a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.562915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.562892 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-metrics-certs\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:41.616161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:41.616097 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqtr\" (UniqueName: \"kubernetes.io/projected/82621e2b-ff5a-45b2-b468-5565a05f7f0a-kube-api-access-prqtr\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:42.071884 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:42.071840 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:42.074277 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:42.074255 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:42.251951 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:42.251911 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:42.618503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:42.618480 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j"] Oct 31 19:38:42.620650 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:38:42.620626 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82621e2b_ff5a_45b2_b468_5565a05f7f0a.slice/crio-efb26071488741532830d1738128d1d6dfb061fda7d38dca2522eab23d6b4f01 WatchSource:0}: Error finding container efb26071488741532830d1738128d1d6dfb061fda7d38dca2522eab23d6b4f01: Status 404 returned error can't find the container with id efb26071488741532830d1738128d1d6dfb061fda7d38dca2522eab23d6b4f01 Oct 31 19:38:43.608861 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:43.608827 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" event={"ID":"82621e2b-ff5a-45b2-b468-5565a05f7f0a","Type":"ContainerStarted","Data":"a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357"} Oct 31 19:38:43.609052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:43.608885 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" event={"ID":"82621e2b-ff5a-45b2-b468-5565a05f7f0a","Type":"ContainerStarted","Data":"efb26071488741532830d1738128d1d6dfb061fda7d38dca2522eab23d6b4f01"} Oct 31 19:38:43.609052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:43.608909 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:43.737020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:43.736892 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" podStartSLOduration=1.880627284 podStartE2EDuration="2.736879462s" totalImagesPullingTime="856.252178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:43.736262155 +0000 UTC m=+1294.976485285" watchObservedRunningTime="2026-10-31 19:38:43.736879462 +0000 UTC m=+1294.977102617" Oct 31 19:38:49.620299 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:49.620270 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 19:38:54.693618 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:54.693571 6481 generic.go:356] "Generic (PLEG): container finished" podID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerID="57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138" exitCode=0 Oct 31 19:38:54.694091 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:54.693637 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" event={"ID":"8d75b9d6-fe3f-4b0e-afde-d3995902d013","Type":"ContainerDied","Data":"57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138"} Oct 31 19:38:54.694091 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:54.694073 6481 scope.go:122] "RemoveContainer" containerID="57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138" Oct 31 19:38:54.695529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:54.695502 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:54.695600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:54.695573 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:54.695635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:54.695603 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:38:55.722448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:55.722399 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" event={"ID":"8d75b9d6-fe3f-4b0e-afde-d3995902d013","Type":"ContainerStarted","Data":"0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1"} Oct 31 19:38:56.615513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:56.615477 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:38:56.615824 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:56.615801 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:38:56.615883 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:56.615859 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 19:38:57.325294 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:38:57.325256 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:38:57.619108 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:57.617663 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:38:57.619108 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:57.617722 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 19:38:58.615920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:58.615876 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:38:58.616449 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:58.615943 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 19:38:59.615358 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:59.615322 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:38:59.615538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:38:59.615374 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 19:39:00.615416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.615380 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:39:00.615821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.615433 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 19:39:00.760979 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.760896 6481 generic.go:356] "Generic (PLEG): container finished" podID="b383e843-8685-4714-b21a-ded1019edf5b" containerID="7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213" exitCode=0 Oct 31 19:39:00.761182 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.760974 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" event={"ID":"b383e843-8685-4714-b21a-ded1019edf5b","Type":"ContainerDied","Data":"7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213"} Oct 31 19:39:00.761416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.761398 6481 scope.go:122] "RemoveContainer" containerID="7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213" Oct 31 19:39:00.762659 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.762631 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:39:00.762748 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.762705 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:39:00.762748 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:00.762738 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:39:01.619825 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:01.619797 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:39:01.620249 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:01.620090 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:39:01.620845 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:01.620825 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 19:39:01.793702 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:01.793671 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" event={"ID":"b383e843-8685-4714-b21a-ded1019edf5b","Type":"ContainerStarted","Data":"259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4"} Oct 31 19:39:02.061573 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:02.061540 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:39:02.062512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:02.062493 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:39:02.062597 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:02.062534 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 19:39:02.218359 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:02.218327 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:39:03.062307 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:03.062274 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:39:03.062714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:03.062326 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 19:39:04.061360 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:04.061322 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:39:04.061571 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:04.061384 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 19:39:05.061371 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:05.061339 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:39:05.061777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:05.061394 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 19:39:06.062560 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:06.062522 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:39:06.062959 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:06.062588 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 19:39:07.065854 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:07.065824 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:39:07.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:39:07.066808 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:39:27.324353 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:39:27.324313 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:39:57.323530 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:39:57.323455 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:40:27.324491 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:40:27.324455 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:40:57.324241 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:40:57.324202 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:41:09.022619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:09.022588 6481 scope.go:122] "RemoveContainer" containerID="8be3148834d006a55d39b4ea873acd88a1e71828a60efded36f2e100128a9fc5" Oct 31 19:41:27.324662 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:41:27.324576 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:41:45.447557 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:45.447526 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg"] Oct 31 19:41:45.448183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:45.447744 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" containerID="cri-o://b3d22035b8ff86bfded35602808ec67736d760f116defd646a3574ecbe64d650" gracePeriod=30 Oct 31 19:41:45.493379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:45.493348 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn"] Oct 31 19:41:45.493619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:45.493594 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" containerID="cri-o://ac9d0933e77d47dcb2c650191b79db1a6e7b4bf5b68b51e42a9603c61c2e19de" gracePeriod=30 Oct 31 19:41:45.495185 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:45.495161 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv"] Oct 31 19:41:45.495330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:45.495313 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" containerID="cri-o://6795cd00de576e3bb54322853fe22d2855b073741fb069f75339fe3a416dd220" gracePeriod=30 Oct 31 19:41:49.930465 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:49.930424 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dxgfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:41:49.930838 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:49.930483 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:41:50.724139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.724070 6481 generic.go:356] "Generic (PLEG): container finished" podID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerID="6795cd00de576e3bb54322853fe22d2855b073741fb069f75339fe3a416dd220" exitCode=0 Oct 31 19:41:50.724301 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.724139 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" event={"ID":"5451f44e-6729-4371-bd5c-2c6d991ab356","Type":"ContainerDied","Data":"6795cd00de576e3bb54322853fe22d2855b073741fb069f75339fe3a416dd220"} Oct 31 19:41:50.725978 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.725954 6481 generic.go:356] "Generic (PLEG): container finished" podID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerID="ac9d0933e77d47dcb2c650191b79db1a6e7b4bf5b68b51e42a9603c61c2e19de" exitCode=0 Oct 31 19:41:50.726124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.726043 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" event={"ID":"abab2ca3-aa23-4dcf-ab92-d01fd1002309","Type":"ContainerDied","Data":"ac9d0933e77d47dcb2c650191b79db1a6e7b4bf5b68b51e42a9603c61c2e19de"} Oct 31 19:41:50.727897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.727875 6481 generic.go:356] "Generic (PLEG): container finished" podID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerID="b3d22035b8ff86bfded35602808ec67736d760f116defd646a3574ecbe64d650" exitCode=0 Oct 31 19:41:50.728026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.727907 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" event={"ID":"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31","Type":"ContainerDied","Data":"b3d22035b8ff86bfded35602808ec67736d760f116defd646a3574ecbe64d650"} Oct 31 19:41:50.758542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.758272 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:41:50.801607 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.801582 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:41:50.826755 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.826731 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:41:50.930095 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930062 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istiod-ca-cert\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.930095 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930099 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-socket\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.930351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930133 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-kube-api-access-q557m\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-kube-api-access-q557m\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.930351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930166 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-kube-api-access-shghl\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-kube-api-access-shghl\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.930351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930209 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-podinfo\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.930351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930243 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-certs\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.930351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930280 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-token\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-token\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.930351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930323 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-podinfo\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.930351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930350 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-data\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930388 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-socket" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930403 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istiod-ca-cert\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930439 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5451f44e-6729-4371-bd5c-2c6d991ab356-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5451f44e-6729-4371-bd5c-2c6d991ab356-istiod-ca-cert\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930478 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-podinfo\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930510 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-socket\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930549 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-token\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-token\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930575 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-certs\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930601 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-credential-socket\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930649 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-certs\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930693 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-credential-socket\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930727 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-credential-socket\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930759 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-envoy\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930795 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-envoy\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930866 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-kube-api-access-f846q\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-kube-api-access-f846q\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930918 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-data\") pod \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\" (UID: \"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930944 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-data\") pod \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\" (UID: \"abab2ca3-aa23-4dcf-ab92-d01fd1002309\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931018 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-socket\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.931145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931100 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-envoy\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.932107 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931168 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-token\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-token\") pod \"5451f44e-6729-4371-bd5c-2c6d991ab356\" (UID: \"5451f44e-6729-4371-bd5c-2c6d991ab356\") " Oct 31 19:41:50.932107 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931742 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930474 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istiod-ca-cert" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930839 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-certs" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.930955 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5451f44e-6729-4371-bd5c-2c6d991ab356-istiod-ca-cert" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931247 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-certs" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931419 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-socket" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931615 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istiod-ca-cert" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931879 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-data" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.931897 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-data" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.932101 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-data" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.932266 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-certs" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.932411 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-credential-socket" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.932753 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-credential-socket" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.932778 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-credential-socket" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.933191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.932785 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-socket" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.934244 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.933425 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-token" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:50.934538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.934510 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-kube-api-access-shghl" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "kube-api-access-shghl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:50.934619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.934582 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-token" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:50.934942 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.934909 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-kube-api-access-q557m" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "kube-api-access-q557m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:50.935043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.935030 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-kube-api-access-f846q" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "kube-api-access-f846q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:50.935241 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.935215 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-token" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:50.935336 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.935314 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-podinfo" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:50.935436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.935418 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-podinfo" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:50.935546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.935515 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-podinfo" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:50.936067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.936046 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-envoy" pod "5451f44e-6729-4371-bd5c-2c6d991ab356" (UID: "5451f44e-6729-4371-bd5c-2c6d991ab356"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.936447 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.936419 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-envoy" pod "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" (UID: "0bffa3ac-6da4-442b-922a-6ef7cc1a5f31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:50.936803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:50.936784 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-envoy" pod "abab2ca3-aa23-4dcf-ab92-d01fd1002309" (UID: "abab2ca3-aa23-4dcf-ab92-d01fd1002309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:51.033807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033717 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f846q\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-kube-api-access-f846q\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.033807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033753 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.033807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033766 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.033807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033779 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.033807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033792 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.033807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033804 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033817 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033829 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q557m\" (UniqueName: \"kubernetes.io/projected/5451f44e-6729-4371-bd5c-2c6d991ab356-kube-api-access-q557m\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033843 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shghl\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-kube-api-access-shghl\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033858 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033870 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033882 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033897 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033910 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033924 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033935 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5451f44e-6729-4371-bd5c-2c6d991ab356-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033946 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033960 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.033971 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.034009 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.034022 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.034035 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.034049 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5451f44e-6729-4371-bd5c-2c6d991ab356-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.034061 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.034076 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.034202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.034090 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab2ca3-aa23-4dcf-ab92-d01fd1002309-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:51.733988 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.733895 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" Oct 31 19:41:51.733988 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.733928 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn" event={"ID":"abab2ca3-aa23-4dcf-ab92-d01fd1002309","Type":"ContainerDied","Data":"19ca592d0fa56f8c2d30ec62d331a6b3435fef2afcb27b9f1437870685487395"} Oct 31 19:41:51.734259 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.734015 6481 scope.go:122] "RemoveContainer" containerID="ac9d0933e77d47dcb2c650191b79db1a6e7b4bf5b68b51e42a9603c61c2e19de" Oct 31 19:41:51.735555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.735531 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" event={"ID":"0bffa3ac-6da4-442b-922a-6ef7cc1a5f31","Type":"ContainerDied","Data":"d7591ec537901f4ac2114534ec6298c1a74417da64c6a9f6d4a7b0f467cb4231"} Oct 31 19:41:51.735741 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.735551 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg" Oct 31 19:41:51.737031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.737013 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" Oct 31 19:41:51.737031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.737016 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv" event={"ID":"5451f44e-6729-4371-bd5c-2c6d991ab356","Type":"ContainerDied","Data":"f78ecf3a5d2f924d2d7a62714aeea3d84ede4c2f83085f4d761e2f485f936a2c"} Oct 31 19:41:51.745390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.745369 6481 scope.go:122] "RemoveContainer" containerID="b3d22035b8ff86bfded35602808ec67736d760f116defd646a3574ecbe64d650" Oct 31 19:41:51.753466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.753441 6481 scope.go:122] "RemoveContainer" containerID="6795cd00de576e3bb54322853fe22d2855b073741fb069f75339fe3a416dd220" Oct 31 19:41:51.887642 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.887602 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv"] Oct 31 19:41:51.934007 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.933970 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dxgfv"] Oct 31 19:41:51.993289 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.993212 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg"] Oct 31 19:41:51.997620 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:51.997597 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sf8qg"] Oct 31 19:41:52.108524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:52.108487 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn"] Oct 31 19:41:52.113438 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:52.113408 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-9qngn"] Oct 31 19:41:52.878299 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:52.878265 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" path="/var/lib/kubelet/pods/0bffa3ac-6da4-442b-922a-6ef7cc1a5f31/volumes" Oct 31 19:41:52.878709 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:52.878696 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" path="/var/lib/kubelet/pods/5451f44e-6729-4371-bd5c-2c6d991ab356/volumes" Oct 31 19:41:52.879138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:41:52.879124 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" path="/var/lib/kubelet/pods/abab2ca3-aa23-4dcf-ab92-d01fd1002309/volumes" Oct 31 19:41:57.324774 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:41:57.324734 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:42:02.034161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034118 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr"] Oct 31 19:42:02.034777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034755 6481 cpu_manager.go:414] "containerMap: removing container" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" Oct 31 19:42:02.034870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034777 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" Oct 31 19:42:02.034870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034806 6481 cpu_manager.go:414] "containerMap: removing container" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" Oct 31 19:42:02.034870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034814 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" Oct 31 19:42:02.034870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034846 6481 cpu_manager.go:414] "containerMap: removing container" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" Oct 31 19:42:02.034870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034854 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" Oct 31 19:42:02.035134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034960 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bffa3ac-6da4-442b-922a-6ef7cc1a5f31" containerName="istio-proxy" Oct 31 19:42:02.035134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034974 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="abab2ca3-aa23-4dcf-ab92-d01fd1002309" containerName="istio-proxy" Oct 31 19:42:02.035134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.034988 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5451f44e-6729-4371-bd5c-2c6d991ab356" containerName="istio-proxy" Oct 31 19:42:02.037079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.037058 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.042861 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.042835 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f34cffd-1e40-424d-9071-99e2595b4ae6-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.042984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.042872 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.042984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.042902 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdml\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-kube-api-access-4sdml\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.043128 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.043053 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.043128 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.043122 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.043227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.043163 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.043280 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.043236 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.043347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.043284 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.043410 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.043377 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.085982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.085954 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-662be1-openshift-default-dockercfg-bwkr4\"" Oct 31 19:42:02.134020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.133973 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr"] Oct 31 19:42:02.143942 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.143912 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.143958 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144008 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144302 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f34cffd-1e40-424d-9071-99e2595b4ae6-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144402 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144435 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdml\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-kube-api-access-4sdml\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144445 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144472 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144500 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144538 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144580 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.144915 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.145243 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.145636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.145330 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f34cffd-1e40-424d-9071-99e2595b4ae6-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.146894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.146868 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.147126 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.147108 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.198233 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.198201 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.198233 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.198218 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdml\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-kube-api-access-4sdml\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.349472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.349390 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:02.764195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.764164 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr"] Oct 31 19:42:02.766619 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:42:02.766590 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f34cffd_1e40_424d_9071_99e2595b4ae6.slice/crio-8aa8f199b519e8445dbd121ed5eac5175a67af82d481ac3811bf24eb92900d2a WatchSource:0}: Error finding container 8aa8f199b519e8445dbd121ed5eac5175a67af82d481ac3811bf24eb92900d2a: Status 404 returned error can't find the container with id 8aa8f199b519e8445dbd121ed5eac5175a67af82d481ac3811bf24eb92900d2a Oct 31 19:42:02.769169 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.769137 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:42:02.769256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.769204 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:42:02.769256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.769235 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:42:02.796056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:02.796026 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" event={"ID":"3f34cffd-1e40-424d-9071-99e2595b4ae6","Type":"ContainerStarted","Data":"8aa8f199b519e8445dbd121ed5eac5175a67af82d481ac3811bf24eb92900d2a"} Oct 31 19:42:03.802960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:03.802926 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" event={"ID":"3f34cffd-1e40-424d-9071-99e2595b4ae6","Type":"ContainerStarted","Data":"62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75"} Oct 31 19:42:03.915371 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:03.915294 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" podStartSLOduration=2.915273199 podStartE2EDuration="2.915273199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:03.911655544 +0000 UTC m=+1495.151878672" watchObservedRunningTime="2026-10-31 19:42:03.915273199 +0000 UTC m=+1495.155496330" Oct 31 19:42:04.350586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:04.350551 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:04.352157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:04.352131 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 19:42:04.352268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:04.352181 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 19:42:05.350627 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:05.350590 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 19:42:05.351024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:05.350643 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 19:42:06.350725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:06.350686 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 19:42:06.351259 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:06.350756 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 19:42:07.350347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:07.350312 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 19:42:07.350503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:07.350366 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 19:42:08.350016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:08.349966 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 19:42:08.350425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:08.350035 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 19:42:09.434024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:09.433904 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:09.435153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:09.435123 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:09.436098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:09.436074 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 19:42:15.916784 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:15.916749 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2"] Oct 31 19:42:15.920626 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:15.920607 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:15.968976 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:15.968942 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b5155c-openshift-default-dockercfg-d5fhs\"" Oct 31 19:42:16.011577 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011541 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011577 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011575 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzc68\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-kube-api-access-mzc68\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011611 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011708 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011767 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011828 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011865 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011935 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011892 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.011967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.011950 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.017250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.017219 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2"] Oct 31 19:42:16.113545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113506 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113546 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc68\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-kube-api-access-mzc68\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113574 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113607 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113653 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113681 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113723 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113762 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.113795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.113789 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.114097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.114007 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.114393 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.114358 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.114393 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.114387 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.114558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.114533 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.114607 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.114593 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.116100 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.116077 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.116191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.116081 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.168041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.167948 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.168334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.168318 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzc68\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-kube-api-access-mzc68\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.233075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.233039 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:16.654415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.654377 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2"] Oct 31 19:42:16.725715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.725283 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:42:16.725715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.725364 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:42:16.725715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.725393 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:42:16.893713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.893678 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" event={"ID":"d6e80273-92b8-45e1-9b49-63aa625fd4ca","Type":"ContainerStarted","Data":"0526a85e8768b187e3e068f603aa67e6ce4f814641ef8d9889dcd7e3181cf4e7"} Oct 31 19:42:16.893957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:16.893736 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" event={"ID":"d6e80273-92b8-45e1-9b49-63aa625fd4ca","Type":"ContainerStarted","Data":"f485923cc1bb3593020f3f018ff3b60dae08ed99af06685633c92aa69d12acd0"} Oct 31 19:42:17.009841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:17.009778 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podStartSLOduration=2.00975844 podStartE2EDuration="2.00975844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:17.007684198 +0000 UTC m=+1508.247907344" watchObservedRunningTime="2026-10-31 19:42:17.00975844 +0000 UTC m=+1508.249981587" Oct 31 19:42:17.235396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:17.235201 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:17.235576 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:17.235513 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 19:42:17.235653 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:17.235575 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 19:42:18.234284 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:18.234250 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 19:42:18.234653 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:18.234300 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 19:42:19.233680 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:19.233645 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 19:42:19.233853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:19.233695 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 19:42:20.233393 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:20.233354 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 19:42:20.233770 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:20.233407 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 19:42:21.233432 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:21.233395 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 19:42:21.233833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:21.233451 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 19:42:22.233899 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:22.233863 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 19:42:22.234344 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:22.233926 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 19:42:23.234042 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:23.233977 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 19:42:23.234564 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:23.234061 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 19:42:24.238429 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:24.238397 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:24.238905 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:24.238680 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:24.239500 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:42:24.239484 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:42:27.324595 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:42:27.324558 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:42:57.324635 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:42:57.324550 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:43:01.030050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.030013 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd"] Oct 31 19:43:01.033740 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.033721 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.076343 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.076314 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd"] Oct 31 19:43:01.079843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.079821 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27"] Oct 31 19:43:01.079976 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.079934 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.083257 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.083234 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.122644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.122611 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd"] Oct 31 19:43:01.131966 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.131936 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.132146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.131981 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d5609-fe3b-46eb-8f30-91211512a1f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.132146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132110 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.132263 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132156 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.132263 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132185 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.132263 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132233 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgzvj\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-kube-api-access-wgzvj\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.132384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132363 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.132444 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132423 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fztn\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-kube-api-access-2fztn\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.132513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132499 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.132599 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132582 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.132655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132638 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsd5z\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-kube-api-access-qsd5z\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.132710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132677 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.132775 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132750 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.132816 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132793 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.132854 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132835 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.132903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132891 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.133051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132951 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.133051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.132977 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.133198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133062 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.133198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133095 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.133198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133129 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.133198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133164 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.133394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133200 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19954c3d-e292-4748-bf7b-16f3cb235fd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.133394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133231 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.133394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133258 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.133394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133297 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.133394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.133330 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1cc5-3044-4453-8c23-4df1fc80b673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.137942 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.137919 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27"] Oct 31 19:43:01.140638 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.140616 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd"] Oct 31 19:43:01.234655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234617 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.234894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234664 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.234894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234691 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.234894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234727 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.234894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234772 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19954c3d-e292-4748-bf7b-16f3cb235fd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.234894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234805 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.234894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234831 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.234894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234872 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234903 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1cc5-3044-4453-8c23-4df1fc80b673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234931 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234957 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d5609-fe3b-46eb-8f30-91211512a1f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.234986 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235053 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235081 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235108 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgzvj\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-kube-api-access-wgzvj\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235149 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235175 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fztn\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-kube-api-access-2fztn\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235225 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235310 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235289 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235343 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235388 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsd5z\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-kube-api-access-qsd5z\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235456 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235225 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235643 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235695 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235725 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235725 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235775 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235810 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235829 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235849 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235871 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.235906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235889 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1cc5-3044-4453-8c23-4df1fc80b673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.236600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.235981 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.236600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.236048 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19954c3d-e292-4748-bf7b-16f3cb235fd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.236600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.236263 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.236600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.236349 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.236600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.236420 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.236876 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.236617 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.236934 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.236911 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.237293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.237195 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d5609-fe3b-46eb-8f30-91211512a1f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.237293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.237282 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.238761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.238721 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.239111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.239082 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.239269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.239240 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.239475 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.239456 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.239542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.239479 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.239591 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.239575 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.290046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.289942 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsd5z\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-kube-api-access-qsd5z\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.290046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.290015 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.290261 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.290098 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fztn\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-kube-api-access-2fztn\") pod \"maas-default-gateway-openshift-default-58d898bd55-czjbd\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.290370 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.290352 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6x5qd\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.290531 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.290514 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.290984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.290966 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgzvj\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-kube-api-access-wgzvj\") pod \"maas-default-gateway-openshift-default-58d898bd55-xkt27\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.349516 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.349480 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:01.394195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.394158 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:01.400269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.400243 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:01.779014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.777876 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd"] Oct 31 19:43:01.814457 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:43:01.814427 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857f1cc5_3044_4453_8c23_4df1fc80b673.slice/crio-6646f4a1277d21d37361ea0d9121444859d85bff2ff8f9ce42f3d84257e19f06 WatchSource:0}: Error finding container 6646f4a1277d21d37361ea0d9121444859d85bff2ff8f9ce42f3d84257e19f06: Status 404 returned error can't find the container with id 6646f4a1277d21d37361ea0d9121444859d85bff2ff8f9ce42f3d84257e19f06 Oct 31 19:43:01.817227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.817194 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.817298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.817263 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.817298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.817292 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.826302 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:43:01.826277 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25d5609_fe3b_46eb_8f30_91211512a1f2.slice/crio-0412596d25632924b87385747a31255cdb0d5fbbd10eeac45bc826e7a086f667 WatchSource:0}: Error finding container 0412596d25632924b87385747a31255cdb0d5fbbd10eeac45bc826e7a086f667: Status 404 returned error can't find the container with id 0412596d25632924b87385747a31255cdb0d5fbbd10eeac45bc826e7a086f667 Oct 31 19:43:01.827467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.827443 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd"] Oct 31 19:43:01.828710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.828682 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.828826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.828738 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.828826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.828778 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.872412 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.871603 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27"] Oct 31 19:43:01.879072 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:43:01.878949 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19954c3d_e292_4748_bf7b_16f3cb235fd0.slice/crio-be69c0f7c11a9ac6cc878451ee874ceb63d037e98555bce3526b7a2942df850f WatchSource:0}: Error finding container be69c0f7c11a9ac6cc878451ee874ceb63d037e98555bce3526b7a2942df850f: Status 404 returned error can't find the container with id be69c0f7c11a9ac6cc878451ee874ceb63d037e98555bce3526b7a2942df850f Oct 31 19:43:01.881621 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.881591 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.881682 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.881669 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:01.881722 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:01.881709 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:43:02.164347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.164250 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" event={"ID":"19954c3d-e292-4748-bf7b-16f3cb235fd0","Type":"ContainerStarted","Data":"c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b"} Oct 31 19:43:02.164347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.164312 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" event={"ID":"19954c3d-e292-4748-bf7b-16f3cb235fd0","Type":"ContainerStarted","Data":"be69c0f7c11a9ac6cc878451ee874ceb63d037e98555bce3526b7a2942df850f"} Oct 31 19:43:02.166226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.166199 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" event={"ID":"a25d5609-fe3b-46eb-8f30-91211512a1f2","Type":"ContainerStarted","Data":"838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9"} Oct 31 19:43:02.166377 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.166249 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" event={"ID":"a25d5609-fe3b-46eb-8f30-91211512a1f2","Type":"ContainerStarted","Data":"0412596d25632924b87385747a31255cdb0d5fbbd10eeac45bc826e7a086f667"} Oct 31 19:43:02.168258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.168231 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" event={"ID":"857f1cc5-3044-4453-8c23-4df1fc80b673","Type":"ContainerStarted","Data":"43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739"} Oct 31 19:43:02.168404 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.168279 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" event={"ID":"857f1cc5-3044-4453-8c23-4df1fc80b673","Type":"ContainerStarted","Data":"6646f4a1277d21d37361ea0d9121444859d85bff2ff8f9ce42f3d84257e19f06"} Oct 31 19:43:02.275934 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.274661 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podStartSLOduration=1.274640572 podStartE2EDuration="1.274640572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:02.274190312 +0000 UTC m=+1553.514413441" watchObservedRunningTime="2026-10-31 19:43:02.274640572 +0000 UTC m=+1553.514863702" Oct 31 19:43:02.350552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.350511 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:02.352045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.352013 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:02.352194 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.352079 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:02.386879 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.386825 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podStartSLOduration=1.386809835 podStartE2EDuration="1.386809835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:02.386320046 +0000 UTC m=+1553.626543187" watchObservedRunningTime="2026-10-31 19:43:02.386809835 +0000 UTC m=+1553.627032964" Oct 31 19:43:02.413051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.399215 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:02.413051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.404556 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:02.413051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.404628 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:02.413051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.404936 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:02.413051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.411422 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:02.413051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:02.411477 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:03.350085 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.350045 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:03.350651 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.350108 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:03.411555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.411520 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:03.411903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.411872 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:03.413856 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.413817 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:03.414070 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.414048 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:03.981480 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.981411 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podStartSLOduration=3.981389077 podStartE2EDuration="3.981389077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:02.507934689 +0000 UTC m=+1553.748157819" watchObservedRunningTime="2026-10-31 19:43:03.981389077 +0000 UTC m=+1555.221612208" Oct 31 19:43:03.984163 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.984134 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46"] Oct 31 19:43:03.998485 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:03.998455 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.054345 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.054301 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46"] Oct 31 19:43:04.097019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.095065 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"distinct-w6-5a4438-kserve-self-signed-certs\"" Oct 31 19:43:04.097019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.095200 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"default-dockercfg-6lxcz\"" Oct 31 19:43:04.097019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.095270 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"kube-root-ca.crt\"" Oct 31 19:43:04.126900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.126216 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a4438\"/\"openshift-service-ca.crt\"" Oct 31 19:43:04.215020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.207921 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-home\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.215020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.207976 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-dshm\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.215020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.208036 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh769\" (UniqueName: \"kubernetes.io/projected/80c67c4b-3638-4895-b372-08d232d4ce2f-kube-api-access-xh769\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.215020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.208122 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-model-cache\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.215020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.208203 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80c67c4b-3638-4895-b372-08d232d4ce2f-tls-certs\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.215020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.208240 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-tmp-dir\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.315076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.309659 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80c67c4b-3638-4895-b372-08d232d4ce2f-tls-certs\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.315076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.309725 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-tmp-dir\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.315076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.309802 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-home\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.315076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.309831 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-dshm\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.315076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.309874 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh769\" (UniqueName: \"kubernetes.io/projected/80c67c4b-3638-4895-b372-08d232d4ce2f-kube-api-access-xh769\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.315076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.309983 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-model-cache\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.315076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.310491 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-model-cache\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.366018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.343480 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-home\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.366018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.343767 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-tmp-dir\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.366018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.352403 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:04.366018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.352464 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:04.414781 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.397435 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:04.414781 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.397555 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:04.464513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.415183 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:04.464513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.415255 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:04.467954 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.467885 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh769\" (UniqueName: \"kubernetes.io/projected/80c67c4b-3638-4895-b372-08d232d4ce2f-kube-api-access-xh769\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.509223 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.481908 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-dshm\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.561677 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.561639 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80c67c4b-3638-4895-b372-08d232d4ce2f-tls-certs\") pod \"distinct-w6-5a4438-kserve-6946c7c7f4-9mq46\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:04.633771 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:04.633474 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:05.242035 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.225786 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46"] Oct 31 19:43:05.250545 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:43:05.250481 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80c67c4b_3638_4895_b372_08d232d4ce2f.slice/crio-b1ea5d7dfadea75a65b95576bf5158a978691362837e93d8e7463487ca64270d WatchSource:0}: Error finding container b1ea5d7dfadea75a65b95576bf5158a978691362837e93d8e7463487ca64270d: Status 404 returned error can't find the container with id b1ea5d7dfadea75a65b95576bf5158a978691362837e93d8e7463487ca64270d Oct 31 19:43:05.356249 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.352359 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:05.356249 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.352527 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:05.447947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.438730 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:05.447947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.442085 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:05.447947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.447083 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:05.447947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.447131 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:05.460420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:05.460367 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" event={"ID":"80c67c4b-3638-4895-b372-08d232d4ce2f","Type":"ContainerStarted","Data":"b1ea5d7dfadea75a65b95576bf5158a978691362837e93d8e7463487ca64270d"} Oct 31 19:43:06.350794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.350744 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:06.350968 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.350812 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:06.397504 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.397382 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:06.397504 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.397444 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:06.401425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.401309 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:06.401425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.401376 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:06.538025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.537045 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" event={"ID":"80c67c4b-3638-4895-b372-08d232d4ce2f","Type":"ContainerStarted","Data":"ebd3ee269c92dd74e4391b85b98fcf03c849cfd0c8683b120066e3016e67bb61"} Oct 31 19:43:06.635191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.635071 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:06.719018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:06.707953 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podStartSLOduration=3.707932355 podStartE2EDuration="3.707932355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:06.70533031 +0000 UTC m=+1557.945553439" watchObservedRunningTime="2026-10-31 19:43:06.707932355 +0000 UTC m=+1557.948155485" Oct 31 19:43:07.350809 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:07.350773 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:07.351009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:07.350822 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:07.399020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:07.396419 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:07.399020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:07.396481 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:07.410018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:07.403350 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:07.410018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:07.403400 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:08.350211 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:08.350169 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:08.350732 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:08.350235 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:08.394778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:08.394736 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:08.394967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:08.394811 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:08.401045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:08.401006 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:08.401227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:08.401086 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:09.350267 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:09.350231 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6x5qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 19:43:09.350803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:09.350288 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 19:43:09.395578 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:09.395530 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-czjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 19:43:09.395789 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:09.395597 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 19:43:09.401643 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:09.401612 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-xkt27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 19:43:09.401821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:09.401676 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 19:43:10.880101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.880063 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:10.882023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.881954 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:10.882851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.882832 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:43:10.939873 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.939839 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:10.940499 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.940476 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:10.941147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.941128 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:43:10.963966 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.963930 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:10.965093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.965065 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:10.965916 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:10.965895 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:43:14.645307 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:14.645277 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:14.698379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:14.698350 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:15.646216 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:15.646191 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:43:27.323690 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:43:27.323653 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:43:54.588293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.588252 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4"] Oct 31 19:43:54.592343 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.592320 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.688841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.688813 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"distinct-w4-7cfafd-kserve-self-signed-certs\"" Oct 31 19:43:54.688841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.688832 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"default-dockercfg-pgfhv\"" Oct 31 19:43:54.688841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.688814 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"kube-root-ca.crt\"" Oct 31 19:43:54.689142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.688857 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"openshift-service-ca.crt\"" Oct 31 19:43:54.691199 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.691179 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4"] Oct 31 19:43:54.725981 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.725948 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-home\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.726168 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.726062 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-tmp-dir\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.726168 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.726142 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdff\" (UniqueName: \"kubernetes.io/projected/cd61e308-a1a7-43df-963c-669ecabe7e77-kube-api-access-rtdff\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.726249 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.726204 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-dshm\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.726249 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.726223 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-model-cache\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.726344 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.726250 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61e308-a1a7-43df-963c-669ecabe7e77-tls-certs\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.826954 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.826917 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-home\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.826969 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-tmp-dir\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.827073 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdff\" (UniqueName: \"kubernetes.io/projected/cd61e308-a1a7-43df-963c-669ecabe7e77-kube-api-access-rtdff\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.827158 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-dshm\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.827186 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-model-cache\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.827229 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61e308-a1a7-43df-963c-669ecabe7e77-tls-certs\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.827377 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-tmp-dir\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.827398 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-home\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.827629 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.827594 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-model-cache\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.829326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.829303 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-dshm\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.829808 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.829789 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61e308-a1a7-43df-963c-669ecabe7e77-tls-certs\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.889857 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.889777 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdff\" (UniqueName: \"kubernetes.io/projected/cd61e308-a1a7-43df-963c-669ecabe7e77-kube-api-access-rtdff\") pod \"distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:54.904558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:54.904532 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:55.291198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:55.291172 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4"] Oct 31 19:43:55.293408 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:43:55.293381 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd61e308_a1a7_43df_963c_669ecabe7e77.slice/crio-f62d1c8ca98a1ad927ffc90400d05c356a63bfecd6172facbf9704135dde2c1b WatchSource:0}: Error finding container f62d1c8ca98a1ad927ffc90400d05c356a63bfecd6172facbf9704135dde2c1b: Status 404 returned error can't find the container with id f62d1c8ca98a1ad927ffc90400d05c356a63bfecd6172facbf9704135dde2c1b Oct 31 19:43:55.900062 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:55.900025 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" event={"ID":"cd61e308-a1a7-43df-963c-669ecabe7e77","Type":"ContainerStarted","Data":"cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45"} Oct 31 19:43:55.900516 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:55.900069 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" event={"ID":"cd61e308-a1a7-43df-963c-669ecabe7e77","Type":"ContainerStarted","Data":"f62d1c8ca98a1ad927ffc90400d05c356a63bfecd6172facbf9704135dde2c1b"} Oct 31 19:43:55.905306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:55.905291 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:43:56.005923 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:43:56.005848 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podStartSLOduration=2.005831262 podStartE2EDuration="2.005831262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:56.005763451 +0000 UTC m=+1607.245986584" watchObservedRunningTime="2026-10-31 19:43:56.005831262 +0000 UTC m=+1607.246054392" Oct 31 19:43:57.324616 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:43:57.324573 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:44:00.174409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.174372 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn"] Oct 31 19:44:00.178266 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.178245 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.231945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.231912 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn"] Oct 31 19:44:00.274588 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.274564 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"openshift-service-ca.crt\"" Oct 31 19:44:00.274776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.274629 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"kube-root-ca.crt\"" Oct 31 19:44:00.274776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.274699 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"default-dockercfg-96g4b\"" Oct 31 19:44:00.274776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.274714 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-570915\"/\"distinct-w5-570915-kserve-self-signed-certs\"" Oct 31 19:44:00.299804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.299553 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-home\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.299804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.299695 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032d683e-b88d-49f9-ad6a-1ce237aee20c-tls-certs\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.300257 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.299832 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-tmp-dir\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.300257 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.299885 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmm5\" (UniqueName: \"kubernetes.io/projected/032d683e-b88d-49f9-ad6a-1ce237aee20c-kube-api-access-nsmm5\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.300257 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.299926 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-dshm\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.300257 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.300037 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-model-cache\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.400658 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.400618 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-model-cache\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.400853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.400676 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-home\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.400853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.400711 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032d683e-b88d-49f9-ad6a-1ce237aee20c-tls-certs\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.400853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.400827 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-tmp-dir\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.401050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.400876 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmm5\" (UniqueName: \"kubernetes.io/projected/032d683e-b88d-49f9-ad6a-1ce237aee20c-kube-api-access-nsmm5\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.401050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.400919 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-dshm\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.401115 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-model-cache\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.401125 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-home\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.401288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.401202 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-tmp-dir\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.403091 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.403065 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-dshm\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.403201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.403142 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032d683e-b88d-49f9-ad6a-1ce237aee20c-tls-certs\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.454248 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.454184 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmm5\" (UniqueName: \"kubernetes.io/projected/032d683e-b88d-49f9-ad6a-1ce237aee20c-kube-api-access-nsmm5\") pod \"distinct-w5-570915-kserve-65954f587c-x8scn\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.491728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.491696 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:00.939701 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:00.939674 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn"] Oct 31 19:44:00.942274 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:44:00.942247 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod032d683e_b88d_49f9_ad6a_1ce237aee20c.slice/crio-49099f764f833de4c5573ffce2f5478a75286efe04edf28aca218a0bcf772c41 WatchSource:0}: Error finding container 49099f764f833de4c5573ffce2f5478a75286efe04edf28aca218a0bcf772c41: Status 404 returned error can't find the container with id 49099f764f833de4c5573ffce2f5478a75286efe04edf28aca218a0bcf772c41 Oct 31 19:44:01.931206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:01.931167 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" event={"ID":"032d683e-b88d-49f9-ad6a-1ce237aee20c","Type":"ContainerStarted","Data":"54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365"} Oct 31 19:44:01.931206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:01.931217 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" event={"ID":"032d683e-b88d-49f9-ad6a-1ce237aee20c","Type":"ContainerStarted","Data":"49099f764f833de4c5573ffce2f5478a75286efe04edf28aca218a0bcf772c41"} Oct 31 19:44:02.041312 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:02.041257 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podStartSLOduration=2.041241903 podStartE2EDuration="2.041241903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:02.039371505 +0000 UTC m=+1613.279594635" watchObservedRunningTime="2026-10-31 19:44:02.041241903 +0000 UTC m=+1613.281465031" Oct 31 19:44:02.492659 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:02.492625 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:04.904757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:04.904722 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:44:04.919219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:04.919190 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:44:04.957553 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:04.957521 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:44:05.670050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.670017 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc"] Oct 31 19:44:05.673922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.673903 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.727074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.727042 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc"] Oct 31 19:44:05.764400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.764371 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-dshm\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.764552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.764436 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e18125-290b-4f97-acd8-0ab00d15f0db-tls-certs\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.764552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.764538 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-model-cache\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.764629 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.764576 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-tmp-dir\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.764629 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.764595 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/90e18125-290b-4f97-acd8-0ab00d15f0db-kube-api-access-cpkpj\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.764757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.764738 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-home\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.770371 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.770350 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"distinct-w3-662be1-kserve-self-signed-certs\"" Oct 31 19:44:05.770466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.770370 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"kube-root-ca.crt\"" Oct 31 19:44:05.770466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.770377 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"openshift-service-ca.crt\"" Oct 31 19:44:05.770466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.770398 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-662be1\"/\"default-dockercfg-4cd7k\"" Oct 31 19:44:05.866454 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866415 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-home\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.866649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866486 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-dshm\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.866649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866576 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e18125-290b-4f97-acd8-0ab00d15f0db-tls-certs\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.866649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866624 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-model-cache\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.866826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866665 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-tmp-dir\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.866826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866691 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/90e18125-290b-4f97-acd8-0ab00d15f0db-kube-api-access-cpkpj\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.866930 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866874 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-home\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.866971 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.866954 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-model-cache\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.867090 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.867065 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-tmp-dir\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.868820 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.868794 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-dshm\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.868933 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.868918 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e18125-290b-4f97-acd8-0ab00d15f0db-tls-certs\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.920532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.920466 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/90e18125-290b-4f97-acd8-0ab00d15f0db-kube-api-access-cpkpj\") pod \"distinct-w3-662be1-kserve-df786bc88-nn7cc\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:05.986378 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:05.986346 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:06.362925 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:06.362905 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc"] Oct 31 19:44:06.365423 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:44:06.365394 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90e18125_290b_4f97_acd8_0ab00d15f0db.slice/crio-f5e6a8ab7f08bacf098592d7ab9cedbb585e7af9e7233914518b55c1119deaba WatchSource:0}: Error finding container f5e6a8ab7f08bacf098592d7ab9cedbb585e7af9e7233914518b55c1119deaba: Status 404 returned error can't find the container with id f5e6a8ab7f08bacf098592d7ab9cedbb585e7af9e7233914518b55c1119deaba Oct 31 19:44:06.959105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:06.959067 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" event={"ID":"90e18125-290b-4f97-acd8-0ab00d15f0db","Type":"ContainerStarted","Data":"48ca44209cc904dc42e6bd2cab385037bfeb046088db447cd110f7d7f6fc762a"} Oct 31 19:44:06.959105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:06.959110 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" event={"ID":"90e18125-290b-4f97-acd8-0ab00d15f0db","Type":"ContainerStarted","Data":"f5e6a8ab7f08bacf098592d7ab9cedbb585e7af9e7233914518b55c1119deaba"} Oct 31 19:44:06.986562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:06.986535 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:07.074282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:07.074236 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podStartSLOduration=2.074221753 podStartE2EDuration="2.074221753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:07.071950273 +0000 UTC m=+1618.312173404" watchObservedRunningTime="2026-10-31 19:44:07.074221753 +0000 UTC m=+1618.314444913" Oct 31 19:44:10.492640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:10.492606 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:10.505295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:10.505270 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:10.993616 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:10.993587 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:44:12.670927 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.670898 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j"] Oct 31 19:44:12.674893 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.674873 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.771320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.771273 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"openshift-service-ca.crt\"" Oct 31 19:44:12.771320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.771304 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"kube-root-ca.crt\"" Oct 31 19:44:12.771940 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.771914 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"default-dockercfg-9626k\"" Oct 31 19:44:12.772146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.772073 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"distinct-w1-b5155c-kserve-self-signed-certs\"" Oct 31 19:44:12.773795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.773775 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j"] Oct 31 19:44:12.857454 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.857406 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-model-cache\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.857729 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.857699 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tmp-dir\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.857862 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.857830 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-dshm\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.857941 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.857912 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-home\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.858102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.858031 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tls-certs\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.858250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.858189 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ll4c\" (UniqueName: \"kubernetes.io/projected/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-kube-api-access-4ll4c\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.959668 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.959598 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ll4c\" (UniqueName: \"kubernetes.io/projected/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-kube-api-access-4ll4c\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.959668 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.959656 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-model-cache\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.959897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.959675 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tmp-dir\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.959897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.959722 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-dshm\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.959897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.959740 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-home\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.959897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.959784 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tls-certs\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.960146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.960108 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tmp-dir\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.960205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.960179 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-model-cache\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.960246 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.960204 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-home\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.962063 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.962040 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-dshm\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:12.962206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:12.962190 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tls-certs\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:13.012913 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:13.012883 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ll4c\" (UniqueName: \"kubernetes.io/projected/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-kube-api-access-4ll4c\") pod \"distinct-w1-b5155c-kserve-754cf54d67-s5k4j\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:13.286613 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:13.286534 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:13.652873 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:13.652847 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j"] Oct 31 19:44:13.654617 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:44:13.654588 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ccebcd0_551f_49f9_9fb3_6957b2c0ac1b.slice/crio-15f163f5b3f7331c216247757b711d0e33a7c4f4ab2192397c6fa65453e6ee7a WatchSource:0}: Error finding container 15f163f5b3f7331c216247757b711d0e33a7c4f4ab2192397c6fa65453e6ee7a: Status 404 returned error can't find the container with id 15f163f5b3f7331c216247757b711d0e33a7c4f4ab2192397c6fa65453e6ee7a Oct 31 19:44:14.002235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:14.002195 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" event={"ID":"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b","Type":"ContainerStarted","Data":"48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b"} Oct 31 19:44:14.002709 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:14.002251 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" event={"ID":"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b","Type":"ContainerStarted","Data":"15f163f5b3f7331c216247757b711d0e33a7c4f4ab2192397c6fa65453e6ee7a"} Oct 31 19:44:14.109444 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:14.109392 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podStartSLOduration=2.109377553 podStartE2EDuration="2.109377553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:14.108038261 +0000 UTC m=+1625.348261388" watchObservedRunningTime="2026-10-31 19:44:14.109377553 +0000 UTC m=+1625.349600682" Oct 31 19:44:14.286675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:14.286606 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:15.987082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:15.987052 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:16.000134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:16.000108 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:16.031945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:16.031919 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:44:17.173984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.173948 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl"] Oct 31 19:44:17.177935 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.177911 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209050 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209117 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa138a5-163c-48aa-9254-02e50743b0b8-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209159 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209278 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209222 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209278 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209244 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209348 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209292 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209348 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209311 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209400 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa138a5-163c-48aa-9254-02e50743b0b8-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.209484 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.209446 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtb4\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-kube-api-access-hxtb4\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.272142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.272088 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-570915-openshift-default-dockercfg-87zft\"" Oct 31 19:44:17.275353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.275324 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl"] Oct 31 19:44:17.310867 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.310841 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa138a5-163c-48aa-9254-02e50743b0b8-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.310915 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtb4\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-kube-api-access-hxtb4\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311029 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311059 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa138a5-163c-48aa-9254-02e50743b0b8-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311174 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311103 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311174 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311147 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311284 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311174 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311284 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311218 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311531 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311576 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311712 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311536 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311745 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.311834 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.311811 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.312119 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.312100 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa138a5-163c-48aa-9254-02e50743b0b8-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.313756 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.313726 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa138a5-163c-48aa-9254-02e50743b0b8-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.313868 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.313846 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.363671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.363636 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtb4\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-kube-api-access-hxtb4\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.363801 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.363692 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.492081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.492049 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:17.906345 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.906318 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl"] Oct 31 19:44:17.908146 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:44:17.908116 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa138a5_163c_48aa_9254_02e50743b0b8.slice/crio-f1329c222a332068fa0ba167295b9359f026107ba7ef42edb0d6de28acf539af WatchSource:0}: Error finding container f1329c222a332068fa0ba167295b9359f026107ba7ef42edb0d6de28acf539af: Status 404 returned error can't find the container with id f1329c222a332068fa0ba167295b9359f026107ba7ef42edb0d6de28acf539af Oct 31 19:44:17.910576 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.910547 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:17.910638 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.910618 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:17.910673 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:17.910649 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:18.034185 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:18.034150 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" event={"ID":"faa138a5-163c-48aa-9254-02e50743b0b8","Type":"ContainerStarted","Data":"399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656"} Oct 31 19:44:18.034569 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:18.034205 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" event={"ID":"faa138a5-163c-48aa-9254-02e50743b0b8","Type":"ContainerStarted","Data":"f1329c222a332068fa0ba167295b9359f026107ba7ef42edb0d6de28acf539af"} Oct 31 19:44:18.144812 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:18.144756 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podStartSLOduration=1.144740837 podStartE2EDuration="1.144740837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:18.143563727 +0000 UTC m=+1629.383786868" watchObservedRunningTime="2026-10-31 19:44:18.144740837 +0000 UTC m=+1629.384963966" Oct 31 19:44:18.492747 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:18.492704 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:18.494673 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:18.494638 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:44:18.494846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:18.494703 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 19:44:19.492407 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:19.492375 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:44:19.492611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:19.492431 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 19:44:20.492638 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:20.492596 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:44:20.493164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:20.492670 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 19:44:21.492518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:21.492484 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:44:21.492697 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:21.492540 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 19:44:22.493302 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:22.493264 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:44:22.493736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:22.493339 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 19:44:23.287414 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:23.287380 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:23.302481 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:23.302451 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:23.493261 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:23.493226 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:44:23.493439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:23.493297 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 19:44:24.086260 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:24.086231 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:44:24.493237 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:24.493203 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 19:44:24.493420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:24.493263 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 19:44:25.496415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:25.496388 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:25.496828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:25.496635 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:25.497277 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:25.497258 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 19:44:27.358310 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:44:27.357984 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:44:47.068363 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.068317 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x"] Oct 31 19:44:47.072292 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.072273 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113002 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.112971 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113023 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113092 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113144 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5lq\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-kube-api-access-hd5lq\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113246 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113304 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113320 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113405 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.113439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.113435 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.170034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.169987 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x"] Oct 31 19:44:47.214748 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.214720 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.214900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.214755 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.214900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.214784 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.214900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.214874 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5lq\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-kube-api-access-hd5lq\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.214915 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.214962 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.214985 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.215105 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.215140 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215323 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.215276 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215323 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.215301 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215417 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.215355 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.215444 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.215716 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.215696 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.217170 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.217149 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.217525 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.217509 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.267787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.267761 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.268235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.268215 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5lq\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-kube-api-access-hd5lq\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.386684 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.386583 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:47.806612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.806586 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x"] Oct 31 19:44:47.807848 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:44:47.807826 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5ff706_dd71_48e4_97e9_c010207cc5e3.slice/crio-2ca5c8a7b8d62d5144960e9341cdf7dc86e33c72b681e28d8a61b836e953c502 WatchSource:0}: Error finding container 2ca5c8a7b8d62d5144960e9341cdf7dc86e33c72b681e28d8a61b836e953c502: Status 404 returned error can't find the container with id 2ca5c8a7b8d62d5144960e9341cdf7dc86e33c72b681e28d8a61b836e953c502 Oct 31 19:44:47.810244 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.810214 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:47.810326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.810275 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:47.810326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:47.810303 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:48.242057 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.242018 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" event={"ID":"6c5ff706-dd71-48e4-97e9-c010207cc5e3","Type":"ContainerStarted","Data":"06d56de779a5f12504badbf4b23a481f4572b0af34a79f0f477a8efd9ddbaa17"} Oct 31 19:44:48.242420 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.242081 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" event={"ID":"6c5ff706-dd71-48e4-97e9-c010207cc5e3","Type":"ContainerStarted","Data":"2ca5c8a7b8d62d5144960e9341cdf7dc86e33c72b681e28d8a61b836e953c502"} Oct 31 19:44:48.347203 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.347170 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn"] Oct 31 19:44:48.351047 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.351028 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.352577 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.352528 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podStartSLOduration=1.352514289 podStartE2EDuration="1.352514289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:48.352084873 +0000 UTC m=+1659.592308003" watchObservedRunningTime="2026-10-31 19:44:48.352514289 +0000 UTC m=+1659.592737420" Oct 31 19:44:48.386935 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.386900 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:48.388409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.388385 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:44:48.388530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.388450 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 19:44:48.415926 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.415895 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn"] Oct 31 19:44:48.429334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429295 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47b6\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-kube-api-access-q47b6\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429494 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429349 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429492 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429623 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429600 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429668 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429734 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429812 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429782 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429896 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.429967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.429951 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.531461 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531362 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.531639 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531473 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47b6\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-kube-api-access-q47b6\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.531639 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531493 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.531639 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531517 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.531639 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531580 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.531920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531640 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.531920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531669 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.532074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.531955 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.532074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.532024 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.532074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.532070 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.532231 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.532100 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.532361 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.532338 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.532524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.532502 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.532933 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.532912 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.533983 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.533963 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.534268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.534243 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.596439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.596407 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47b6\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-kube-api-access-q47b6\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.597485 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.597457 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:48.687019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:48.686227 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:49.103509 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.103474 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn"] Oct 31 19:44:49.104819 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:44:49.104792 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7850a1_eb98_4f34_b9c3_b0eae9bd8775.slice/crio-618d2c31c2e79f1e27e13167df381ba51ade4ce33e5d19bbd5251c2b2e48ce5d WatchSource:0}: Error finding container 618d2c31c2e79f1e27e13167df381ba51ade4ce33e5d19bbd5251c2b2e48ce5d: Status 404 returned error can't find the container with id 618d2c31c2e79f1e27e13167df381ba51ade4ce33e5d19bbd5251c2b2e48ce5d Oct 31 19:44:49.107390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.107360 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:49.107456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.107422 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:49.107456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.107451 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:44:49.251849 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.251806 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" event={"ID":"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775","Type":"ContainerStarted","Data":"5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8"} Oct 31 19:44:49.252300 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.251861 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" event={"ID":"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775","Type":"ContainerStarted","Data":"618d2c31c2e79f1e27e13167df381ba51ade4ce33e5d19bbd5251c2b2e48ce5d"} Oct 31 19:44:49.361514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.361405 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podStartSLOduration=1.361388107 podStartE2EDuration="1.361388107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:49.359574347 +0000 UTC m=+1660.599797476" watchObservedRunningTime="2026-10-31 19:44:49.361388107 +0000 UTC m=+1660.601611235" Oct 31 19:44:49.390982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.390429 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:44:49.390982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.390490 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 19:44:49.690164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:49.690080 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:50.394018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:50.393471 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:44:50.394018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:50.393554 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 19:44:50.689850 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:50.689759 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:44:50.690130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:50.690099 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 19:44:51.389830 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:51.389353 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:44:51.389830 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:51.389429 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 19:44:51.691503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:51.691406 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:44:51.692136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:51.692104 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 19:44:52.396481 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:52.396441 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:44:52.396778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:52.396748 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 19:44:52.692074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:52.691422 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:44:52.692074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:52.691485 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 19:44:53.390020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:53.388843 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:44:53.390020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:53.388904 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 19:44:53.691472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:53.691374 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:44:53.691750 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:53.691722 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 19:44:54.389425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:54.389387 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 19:44:54.390046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:54.390011 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 19:44:54.691445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:54.691357 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:44:54.691445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:54.691422 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 19:44:55.392973 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:55.392934 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:55.393790 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:55.393768 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:55.394744 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:55.394718 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:44:55.687801 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:55.687713 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:44:55.687801 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:55.687775 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 19:44:56.691833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:56.691808 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:56.692314 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:56.692234 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:56.692966 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:44:56.692949 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 19:44:57.325162 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:44:57.325129 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:45:00.190138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.190104 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf"] Oct 31 19:45:00.193978 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.193953 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.244782 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.244718 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 19:45:00.278925 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.278884 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80fe344-8734-4d29-9dee-f2825c5dd80b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.279233 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.279201 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccnxg\" (UniqueName: \"kubernetes.io/projected/c80fe344-8734-4d29-9dee-f2825c5dd80b-kube-api-access-ccnxg\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.374919 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.374886 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf"] Oct 31 19:45:00.380960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.380932 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80fe344-8734-4d29-9dee-f2825c5dd80b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.381146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.381123 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccnxg\" (UniqueName: \"kubernetes.io/projected/c80fe344-8734-4d29-9dee-f2825c5dd80b-kube-api-access-ccnxg\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.382251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.381902 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80fe344-8734-4d29-9dee-f2825c5dd80b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.434309 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.434281 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccnxg\" (UniqueName: \"kubernetes.io/projected/c80fe344-8734-4d29-9dee-f2825c5dd80b-kube-api-access-ccnxg\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.508859 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.508822 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:00.879671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.879647 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf"] Oct 31 19:45:00.880873 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:45:00.880844 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80fe344_8734_4d29_9dee_f2825c5dd80b.slice/crio-3c0a54df27123d01504522c949715f26f548ca0f4c136323e80916ca33f2d144 WatchSource:0}: Error finding container 3c0a54df27123d01504522c949715f26f548ca0f4c136323e80916ca33f2d144: Status 404 returned error can't find the container with id 3c0a54df27123d01504522c949715f26f548ca0f4c136323e80916ca33f2d144 Oct 31 19:45:00.882686 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:00.882670 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:45:01.348324 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:01.348283 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" event={"ID":"c80fe344-8734-4d29-9dee-f2825c5dd80b","Type":"ContainerStarted","Data":"3c0a54df27123d01504522c949715f26f548ca0f4c136323e80916ca33f2d144"} Oct 31 19:45:05.375324 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:05.375237 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" event={"ID":"c80fe344-8734-4d29-9dee-f2825c5dd80b","Type":"ContainerStarted","Data":"04be1b0f940bd33bf1537050e01eeccf208afad8e879d9168cb2900ea6c72731"} Oct 31 19:45:05.480941 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:05.480889 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" podStartSLOduration=1.61438552 podStartE2EDuration="5.480876091s" totalImagesPullingTime="3.866490571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:05.47977575 +0000 UTC m=+1676.719998883" watchObservedRunningTime="2026-10-31 19:45:05.480876091 +0000 UTC m=+1676.721099220" Oct 31 19:45:06.382048 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:06.382006 6481 generic.go:356] "Generic (PLEG): container finished" podID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerID="04be1b0f940bd33bf1537050e01eeccf208afad8e879d9168cb2900ea6c72731" exitCode=60 Oct 31 19:45:06.382446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:06.382097 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" event={"ID":"c80fe344-8734-4d29-9dee-f2825c5dd80b","Type":"ContainerDied","Data":"04be1b0f940bd33bf1537050e01eeccf208afad8e879d9168cb2900ea6c72731"} Oct 31 19:45:06.382446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:06.382312 6481 scope.go:122] "RemoveContainer" containerID="04be1b0f940bd33bf1537050e01eeccf208afad8e879d9168cb2900ea6c72731" Oct 31 19:45:07.389157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:07.389122 6481 generic.go:356] "Generic (PLEG): container finished" podID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerID="189fed8399482aeabcc5a52400ae00bb8b67e4458123dc7baff3bc63d653ea9f" exitCode=60 Oct 31 19:45:07.389157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:07.389159 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" event={"ID":"c80fe344-8734-4d29-9dee-f2825c5dd80b","Type":"ContainerDied","Data":"189fed8399482aeabcc5a52400ae00bb8b67e4458123dc7baff3bc63d653ea9f"} Oct 31 19:45:07.389640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:07.389200 6481 scope.go:122] "RemoveContainer" containerID="04be1b0f940bd33bf1537050e01eeccf208afad8e879d9168cb2900ea6c72731" Oct 31 19:45:07.389640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:07.389487 6481 scope.go:122] "RemoveContainer" containerID="189fed8399482aeabcc5a52400ae00bb8b67e4458123dc7baff3bc63d653ea9f" Oct 31 19:45:07.389831 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:45:07.389803 6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf_odh-ai-gateway-infra(c80fe344-8734-4d29-9dee-f2825c5dd80b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" Oct 31 19:45:08.396437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:08.396411 6481 scope.go:122] "RemoveContainer" containerID="189fed8399482aeabcc5a52400ae00bb8b67e4458123dc7baff3bc63d653ea9f" Oct 31 19:45:08.396833 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:45:08.396665 6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf_odh-ai-gateway-infra(c80fe344-8734-4d29-9dee-f2825c5dd80b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" Oct 31 19:45:20.872589 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:20.872546 6481 scope.go:122] "RemoveContainer" containerID="189fed8399482aeabcc5a52400ae00bb8b67e4458123dc7baff3bc63d653ea9f" Oct 31 19:45:21.490056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:21.488891 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" event={"ID":"c80fe344-8734-4d29-9dee-f2825c5dd80b","Type":"ContainerStarted","Data":"ece7e92a3a4b38d59977de86bed0598b4cc79370a16168c60a57cbf3a4ed8991"} Oct 31 19:45:22.496725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:22.496685 6481 generic.go:356] "Generic (PLEG): container finished" podID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerID="ece7e92a3a4b38d59977de86bed0598b4cc79370a16168c60a57cbf3a4ed8991" exitCode=60 Oct 31 19:45:22.497236 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:22.496758 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" event={"ID":"c80fe344-8734-4d29-9dee-f2825c5dd80b","Type":"ContainerDied","Data":"ece7e92a3a4b38d59977de86bed0598b4cc79370a16168c60a57cbf3a4ed8991"} Oct 31 19:45:22.497236 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:22.496842 6481 scope.go:122] "RemoveContainer" containerID="189fed8399482aeabcc5a52400ae00bb8b67e4458123dc7baff3bc63d653ea9f" Oct 31 19:45:22.497236 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:22.497219 6481 scope.go:122] "RemoveContainer" containerID="ece7e92a3a4b38d59977de86bed0598b4cc79370a16168c60a57cbf3a4ed8991" Oct 31 19:45:22.497514 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:45:22.497488 6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf_odh-ai-gateway-infra(c80fe344-8734-4d29-9dee-f2825c5dd80b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" Oct 31 19:45:22.613915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:22.613885 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf"] Oct 31 19:45:23.655954 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:23.655927 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:23.775982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:23.775893 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80fe344-8734-4d29-9dee-f2825c5dd80b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80fe344-8734-4d29-9dee-f2825c5dd80b-maas-api-service-ca\") pod \"c80fe344-8734-4d29-9dee-f2825c5dd80b\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " Oct 31 19:45:23.776185 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:23.776034 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80fe344-8734-4d29-9dee-f2825c5dd80b-kube-api-access-ccnxg\" (UniqueName: \"kubernetes.io/projected/c80fe344-8734-4d29-9dee-f2825c5dd80b-kube-api-access-ccnxg\") pod \"c80fe344-8734-4d29-9dee-f2825c5dd80b\" (UID: \"c80fe344-8734-4d29-9dee-f2825c5dd80b\") " Oct 31 19:45:23.776334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:23.776308 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80fe344-8734-4d29-9dee-f2825c5dd80b-maas-api-service-ca" pod "c80fe344-8734-4d29-9dee-f2825c5dd80b" (UID: "c80fe344-8734-4d29-9dee-f2825c5dd80b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:23.776689 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:23.776668 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80fe344-8734-4d29-9dee-f2825c5dd80b-maas-api-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:23.778511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:23.778486 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80fe344-8734-4d29-9dee-f2825c5dd80b-kube-api-access-ccnxg" pod "c80fe344-8734-4d29-9dee-f2825c5dd80b" (UID: "c80fe344-8734-4d29-9dee-f2825c5dd80b"). InnerVolumeSpecName "kube-api-access-ccnxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:23.878162 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:23.878124 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccnxg\" (UniqueName: \"kubernetes.io/projected/c80fe344-8734-4d29-9dee-f2825c5dd80b-kube-api-access-ccnxg\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:24.511913 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:24.511882 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" event={"ID":"c80fe344-8734-4d29-9dee-f2825c5dd80b","Type":"ContainerDied","Data":"3c0a54df27123d01504522c949715f26f548ca0f4c136323e80916ca33f2d144"} Oct 31 19:45:24.512120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:24.511918 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf" Oct 31 19:45:24.512120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:24.511941 6481 scope.go:122] "RemoveContainer" containerID="ece7e92a3a4b38d59977de86bed0598b4cc79370a16168c60a57cbf3a4ed8991" Oct 31 19:45:24.664016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:24.663972 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf"] Oct 31 19:45:24.710441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:24.710411 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a4438-29891265-csxrf"] Oct 31 19:45:24.882325 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:24.882251 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" path="/var/lib/kubelet/pods/c80fe344-8734-4d29-9dee-f2825c5dd80b/volumes" Oct 31 19:45:27.325246 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:45:27.325211 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:45:30.281444 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.281411 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm"] Oct 31 19:45:30.281931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.281918 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:30.281977 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.281932 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:30.281977 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.281944 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:30.281977 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.281949 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:30.282087 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.282061 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:30.282087 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.282076 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:30.285390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.285375 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.381553 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.381517 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7cfafd\"/\"distinct-2-w4-7cfafd-kserve-self-signed-certs\"" Oct 31 19:45:30.384234 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.384198 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm"] Oct 31 19:45:30.460875 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.460803 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-tmp-dir\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.462636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.462588 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv46q\" (UniqueName: \"kubernetes.io/projected/69df7514-491c-45c4-bb68-2677a787f0da-kube-api-access-rv46q\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.462965 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.462936 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-home\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.463498 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.463476 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-model-cache\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.463727 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.463709 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-dshm\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.463977 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.463952 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69df7514-491c-45c4-bb68-2677a787f0da-tls-certs\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.564964 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.564879 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-tmp-dir\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565154 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.564968 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv46q\" (UniqueName: \"kubernetes.io/projected/69df7514-491c-45c4-bb68-2677a787f0da-kube-api-access-rv46q\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565154 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.565059 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-home\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565154 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.565075 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-model-cache\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565154 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.565114 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-dshm\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565382 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.565170 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69df7514-491c-45c4-bb68-2677a787f0da-tls-certs\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.565399 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-tmp-dir\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565596 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.565566 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-model-cache\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.565861 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.565841 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-home\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.567783 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.567763 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-dshm\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.567894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.567783 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69df7514-491c-45c4-bb68-2677a787f0da-tls-certs\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.617754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.617723 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv46q\" (UniqueName: \"kubernetes.io/projected/69df7514-491c-45c4-bb68-2677a787f0da-kube-api-access-rv46q\") pod \"distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:30.904462 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:30.904378 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:31.270071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:31.270044 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm"] Oct 31 19:45:31.272076 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:45:31.272052 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69df7514_491c_45c4_bb68_2677a787f0da.slice/crio-6360c42cb596c821bff421c63969fb6510859ce5fabc66955e5c3be79c6d9849 WatchSource:0}: Error finding container 6360c42cb596c821bff421c63969fb6510859ce5fabc66955e5c3be79c6d9849: Status 404 returned error can't find the container with id 6360c42cb596c821bff421c63969fb6510859ce5fabc66955e5c3be79c6d9849 Oct 31 19:45:31.553217 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:31.553127 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" event={"ID":"69df7514-491c-45c4-bb68-2677a787f0da","Type":"ContainerStarted","Data":"db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b"} Oct 31 19:45:31.553217 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:31.553174 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" event={"ID":"69df7514-491c-45c4-bb68-2677a787f0da","Type":"ContainerStarted","Data":"6360c42cb596c821bff421c63969fb6510859ce5fabc66955e5c3be79c6d9849"} Oct 31 19:45:31.661476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:31.661420 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podStartSLOduration=1.6614045389999998 podStartE2EDuration="1.661404539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:31.660321678 +0000 UTC m=+1702.900544813" watchObservedRunningTime="2026-10-31 19:45:31.661404539 +0000 UTC m=+1702.901627693" Oct 31 19:45:31.905479 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:31.905392 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:32.114193 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.114161 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c"] Oct 31 19:45:32.114763 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.114748 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:32.114820 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.114764 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:32.114887 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.114875 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80fe344-8734-4d29-9dee-f2825c5dd80b" containerName="cleanup" Oct 31 19:45:32.118333 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.118314 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.215873 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.215804 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c"] Oct 31 19:45:32.294378 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294341 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294489 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294538 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294648 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294707 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294694 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294768 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294756 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294788 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294845 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294807 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.294942 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.294925 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2cc\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-kube-api-access-hd2cc\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396047 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396255 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396098 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396255 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396138 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396255 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396231 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396324 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396358 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396463 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396386 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396519 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396463 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2cc\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-kube-api-access-hd2cc\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396515 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396533 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396552 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396649 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.396801 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.396773 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.397350 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.397322 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.398540 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.398513 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.399113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.399096 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.449303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.449273 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2cc\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-kube-api-access-hd2cc\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.449303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.449300 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:32.732521 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:32.732488 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:33.150704 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.150679 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c"] Oct 31 19:45:33.153487 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:45:33.153458 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9eb013_bc86_47e6_9ae7_f14e49981a6f.slice/crio-3fadfa66e64a15187f2a3f964c9a537668c0738ed590481858f4fba342049ac4 WatchSource:0}: Error finding container 3fadfa66e64a15187f2a3f964c9a537668c0738ed590481858f4fba342049ac4: Status 404 returned error can't find the container with id 3fadfa66e64a15187f2a3f964c9a537668c0738ed590481858f4fba342049ac4 Oct 31 19:45:33.156461 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.156430 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:45:33.156532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.156496 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:45:33.156532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.156524 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:45:33.566184 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.566143 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" event={"ID":"7e9eb013-bc86-47e6-9ae7-f14e49981a6f","Type":"ContainerStarted","Data":"70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97"} Oct 31 19:45:33.566387 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.566212 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" event={"ID":"7e9eb013-bc86-47e6-9ae7-f14e49981a6f","Type":"ContainerStarted","Data":"3fadfa66e64a15187f2a3f964c9a537668c0738ed590481858f4fba342049ac4"} Oct 31 19:45:33.722609 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.722548 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podStartSLOduration=1.722534793 podStartE2EDuration="1.722534793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:33.718905718 +0000 UTC m=+1704.959128848" watchObservedRunningTime="2026-10-31 19:45:33.722534793 +0000 UTC m=+1704.962757922" Oct 31 19:45:33.733513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:33.733490 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:34.734577 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:34.734541 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:45:34.734968 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:34.734591 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 19:45:35.733555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:35.733518 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:45:35.733739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:35.733573 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 19:45:36.732798 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:36.732765 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:45:36.733235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:36.732815 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 19:45:37.733083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:37.733049 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:45:37.733544 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:37.733098 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 19:45:38.733647 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:38.733606 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:45:38.734097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:38.733686 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 19:45:39.733688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:39.733658 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 19:45:39.734093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:39.733713 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 19:45:40.738163 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:40.738133 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:40.738581 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:40.738503 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:40.739248 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:40.739227 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:45:40.904768 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:40.904735 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:40.917395 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:40.917362 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:41.649850 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:45:41.649812 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:45:57.324869 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:45:57.324822 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:46:17.462868 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.462833 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt"] Oct 31 19:46:17.468272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.468246 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.499842 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.499809 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.499869 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cfm\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-kube-api-access-64cfm\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.499956 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.500070 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.500137 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500229 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.500157 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.500254 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.500284 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.500351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.500312 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601366 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601330 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601392 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cfm\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-kube-api-access-64cfm\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601419 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601450 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601502 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601521 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601558 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601593 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.601767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.601634 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.602096 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.602065 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.602224 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.602154 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.602291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.602226 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.602291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.602254 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.602426 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.602408 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.603988 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.603963 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.604242 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.604223 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.657206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.657153 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt"] Oct 31 19:46:17.659328 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.658642 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.659807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.659783 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cfm\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-kube-api-access-64cfm\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:17.781211 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:17.781124 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:18.205815 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:18.205783 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt"] Oct 31 19:46:18.207654 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:46:18.207631 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0eb8290_aa75_4bc5_ba36_4c090b54cc82.slice/crio-956fd475488c24fac67843a11bba3d2f5fd998a523bbcb7e861a0f29d73b61f7 WatchSource:0}: Error finding container 956fd475488c24fac67843a11bba3d2f5fd998a523bbcb7e861a0f29d73b61f7: Status 404 returned error can't find the container with id 956fd475488c24fac67843a11bba3d2f5fd998a523bbcb7e861a0f29d73b61f7 Oct 31 19:46:18.210034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:18.209978 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:46:18.210115 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:18.210064 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:46:18.210115 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:18.210095 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:46:18.901737 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:18.901701 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" event={"ID":"d0eb8290-aa75-4bc5-ba36-4c090b54cc82","Type":"ContainerStarted","Data":"21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f"} Oct 31 19:46:18.902250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:18.901756 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" event={"ID":"d0eb8290-aa75-4bc5-ba36-4c090b54cc82","Type":"ContainerStarted","Data":"956fd475488c24fac67843a11bba3d2f5fd998a523bbcb7e861a0f29d73b61f7"} Oct 31 19:46:19.058961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:19.058909 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podStartSLOduration=2.05889606 podStartE2EDuration="2.05889606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:19.057817954 +0000 UTC m=+1750.298041083" watchObservedRunningTime="2026-10-31 19:46:19.05889606 +0000 UTC m=+1750.299119189" Oct 31 19:46:19.789364 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:19.789328 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:19.803505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:19.798558 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:46:19.803505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:19.798625 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 19:46:20.788018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:20.786349 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:46:20.788018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:20.786411 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 19:46:21.785015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:21.782340 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:46:21.785015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:21.782419 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 19:46:22.783596 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:22.783560 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:46:22.784196 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:22.784166 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 19:46:23.787019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:23.783391 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:46:23.787019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:23.783456 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 19:46:24.782176 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:24.782145 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:46:24.782370 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:24.782203 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 19:46:26.600148 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:26.600114 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:26.600932 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:26.600907 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:26.601762 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:26.601737 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:46:27.324375 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:46:27.324339 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:46:33.276208 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.276172 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv"] Oct 31 19:46:33.280159 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.280140 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.345384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.345344 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-tmp-dir\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.345548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.345485 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-dshm\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.345548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.345530 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f425b-defd-4b57-9389-eea11ec6df18-tls-certs\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.345630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.345551 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77gcf\" (UniqueName: \"kubernetes.io/projected/ae6f425b-defd-4b57-9389-eea11ec6df18-kube-api-access-77gcf\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.345630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.345591 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-model-cache\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.345630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.345619 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-home\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.376804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.376773 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b5155c\"/\"distinct-2-w1-b5155c-kserve-self-signed-certs\"" Oct 31 19:46:33.379298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.379275 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv"] Oct 31 19:46:33.461276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.451171 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-model-cache\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.461654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.461618 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-model-cache\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.461843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.461682 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-home\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.462034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.461986 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-home\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.462256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.462237 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-tmp-dir\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.462526 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.462505 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-dshm\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.462699 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.462684 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f425b-defd-4b57-9389-eea11ec6df18-tls-certs\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.462814 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.462799 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gcf\" (UniqueName: \"kubernetes.io/projected/ae6f425b-defd-4b57-9389-eea11ec6df18-kube-api-access-77gcf\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.475269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.475234 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-tmp-dir\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.511014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.510155 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f425b-defd-4b57-9389-eea11ec6df18-tls-certs\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.512402 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.512370 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-dshm\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.528668 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.528581 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77gcf\" (UniqueName: \"kubernetes.io/projected/ae6f425b-defd-4b57-9389-eea11ec6df18-kube-api-access-77gcf\") pod \"distinct-2-w1-b5155c-kserve-566d684844-bnsjv\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.593223 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.593194 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:33.959961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:33.959936 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv"] Oct 31 19:46:33.961721 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:46:33.961695 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae6f425b_defd_4b57_9389_eea11ec6df18.slice/crio-9e7b28c41c8938c21e75e9beb1bac6992b4502bb5f6fe18e4c115bf05c6e8be0 WatchSource:0}: Error finding container 9e7b28c41c8938c21e75e9beb1bac6992b4502bb5f6fe18e4c115bf05c6e8be0: Status 404 returned error can't find the container with id 9e7b28c41c8938c21e75e9beb1bac6992b4502bb5f6fe18e4c115bf05c6e8be0 Oct 31 19:46:34.016589 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:34.016559 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" event={"ID":"ae6f425b-defd-4b57-9389-eea11ec6df18","Type":"ContainerStarted","Data":"9e7b28c41c8938c21e75e9beb1bac6992b4502bb5f6fe18e4c115bf05c6e8be0"} Oct 31 19:46:35.024414 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:35.024376 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" event={"ID":"ae6f425b-defd-4b57-9389-eea11ec6df18","Type":"ContainerStarted","Data":"5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944"} Oct 31 19:46:35.595121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:35.595085 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:43.593632 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:43.593597 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:43.606339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:43.606313 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:43.714052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:43.713971 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podStartSLOduration=10.713956367 podStartE2EDuration="10.713956367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:35.144261771 +0000 UTC m=+1766.384484901" watchObservedRunningTime="2026-10-31 19:46:43.713956367 +0000 UTC m=+1774.954179498" Oct 31 19:46:44.171256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:44.171231 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:46:56.952179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:56.952120 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:46:56.991134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:56.991093 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:46:56.991695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:56.991662 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 31 19:46:57.026117 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:57.026076 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:46:57.074675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:57.073925 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:46:57.179894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:46:57.179080 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:46:57.370016 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:46:57.366789 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:47:03.363164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.363129 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c"] Oct 31 19:47:03.367503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.367482 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.463598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.463565 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c"] Oct 31 19:47:03.479041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.478983 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479137 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbb9999-8162-4689-9e55-8b39e664322f-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479254 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479186 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479352 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479336 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgx2v\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-kube-api-access-pgx2v\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479399 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479385 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479516 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb9999-8162-4689-9e55-8b39e664322f-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479625 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479607 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479693 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479675 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.479775 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.479761 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.581821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.581787 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.581984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.581843 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.581984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.581904 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.581983 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbb9999-8162-4689-9e55-8b39e664322f-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582036 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582118 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgx2v\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-kube-api-access-pgx2v\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582165 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582229 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb9999-8162-4689-9e55-8b39e664322f-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582281 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582284 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582534 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582614 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582543 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582680 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582664 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.582979 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.582959 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb9999-8162-4689-9e55-8b39e664322f-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.584831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.584802 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.585015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.584984 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbb9999-8162-4689-9e55-8b39e664322f-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.635152 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.635092 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.635367 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.635345 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgx2v\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-kube-api-access-pgx2v\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:03.681064 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:03.681027 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:04.094512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.094489 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c"] Oct 31 19:47:04.095955 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:47:04.095925 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbb9999_8162_4689_9e55_8b39e664322f.slice/crio-902f3e1831e8c87b2e1760e415eb56e0feadb25859fd661713af3568aedc65da WatchSource:0}: Error finding container 902f3e1831e8c87b2e1760e415eb56e0feadb25859fd661713af3568aedc65da: Status 404 returned error can't find the container with id 902f3e1831e8c87b2e1760e415eb56e0feadb25859fd661713af3568aedc65da Oct 31 19:47:04.098233 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.098200 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:04.098305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.098263 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:04.098305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.098289 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:04.350116 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.350031 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" event={"ID":"9bbb9999-8162-4689-9e55-8b39e664322f","Type":"ContainerStarted","Data":"98653b61eeee2b23288bead99a8b63422cab3a5dd1a84060042732c872f857c4"} Oct 31 19:47:04.350116 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.350097 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" event={"ID":"9bbb9999-8162-4689-9e55-8b39e664322f","Type":"ContainerStarted","Data":"902f3e1831e8c87b2e1760e415eb56e0feadb25859fd661713af3568aedc65da"} Oct 31 19:47:04.460113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.460061 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podStartSLOduration=1.460043212 podStartE2EDuration="1.460043212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:04.458034212 +0000 UTC m=+1795.698257339" watchObservedRunningTime="2026-10-31 19:47:04.460043212 +0000 UTC m=+1795.700266341" Oct 31 19:47:04.682810 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:04.682703 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:05.686166 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:05.686131 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:05.686788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:05.686736 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 19:47:06.686017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:06.685401 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:06.686017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:06.685464 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 19:47:07.682826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:07.682378 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:07.682826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:07.682439 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 19:47:08.683767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:08.683331 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:08.683767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:08.683394 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 19:47:09.685028 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:09.684974 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:09.685592 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:09.685561 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 19:47:10.681486 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:10.681445 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 19:47:10.681663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:10.681524 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 19:47:11.686005 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:11.685960 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:11.686375 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:11.686354 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:11.686969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:11.686948 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 19:47:18.226413 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.226380 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn"] Oct 31 19:47:18.231488 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.231471 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303399 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303374 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303402 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78764e47-af66-4b6d-aa36-c47a927b2d3a-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303451 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-kube-api-access-qxp2k\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303616 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303531 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303653 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303624 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303687 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303655 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303731 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303867 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303855 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.303908 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.303878 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.404609 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.404569 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.404791 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.404617 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.404791 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.404658 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.404911 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.404840 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.404911 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.404869 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.404931 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.404956 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78764e47-af66-4b6d-aa36-c47a927b2d3a-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.405036 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-kube-api-access-qxp2k\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.405072 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.405179 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405594 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.405546 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405594 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.405566 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.405796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.405772 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.406041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.406014 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78764e47-af66-4b6d-aa36-c47a927b2d3a-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.407949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.407918 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.408088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.408046 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.428754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.428722 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn"] Oct 31 19:47:18.462607 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.459953 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.462607 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.460389 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-kube-api-access-qxp2k\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.545476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.545392 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:18.799155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.799066 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd"] Oct 31 19:47:18.811182 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.811100 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.814551 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.814520 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk"] Oct 31 19:47:18.818659 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.818634 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.859100 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.859060 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd"] Oct 31 19:47:18.913776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.913723 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.913776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.913780 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.914073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.913865 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.914073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.913931 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.914073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.914045 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.914305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.914282 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.914534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.914513 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.914621 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.914546 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.916081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916054 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.916195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916096 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.916195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916129 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.916195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916175 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.916372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916230 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.916372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916260 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnnt\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-kube-api-access-swnnt\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.916372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916321 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.916372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916346 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:18.916578 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916382 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.916578 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.916555 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcgr\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-kube-api-access-8tcgr\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:18.921094 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.921068 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk"] Oct 31 19:47:18.965082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.965052 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn"] Oct 31 19:47:18.965715 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:47:18.965691 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78764e47_af66_4b6d_aa36_c47a927b2d3a.slice/crio-7d1975db037e7ccb4e04160ce6f91248ffcb3d43e9ba06ce316eb6f866a290fe WatchSource:0}: Error finding container 7d1975db037e7ccb4e04160ce6f91248ffcb3d43e9ba06ce316eb6f866a290fe: Status 404 returned error can't find the container with id 7d1975db037e7ccb4e04160ce6f91248ffcb3d43e9ba06ce316eb6f866a290fe Oct 31 19:47:18.968800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.968769 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:18.968876 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.968832 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:18.968876 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:18.968862 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.017751 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.017722 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.017925 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.017766 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnnt\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-kube-api-access-swnnt\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.017925 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.017798 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.017925 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.017816 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.017925 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.017853 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.017945 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcgr\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-kube-api-access-8tcgr\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018049 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.018212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018081 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018111 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018133 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018184 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018213 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018255 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018269 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018343 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018390 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018459 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018466 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018485 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018488 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-credential-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.018532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018515 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.019098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018570 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.019098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.018714 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-certs\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.019098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.019021 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-data\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.019098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.019043 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.019328 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.019299 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-socket\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.019407 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.019340 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.019407 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.019371 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istiod-ca-cert\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.020963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.020934 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.021189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.021168 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.021265 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.021198 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-envoy\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.021466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.021450 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-podinfo\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.072158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.072071 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.072158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.072137 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcgr\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-kube-api-access-8tcgr\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.072480 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.072451 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnnt\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-kube-api-access-swnnt\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.072859 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.072837 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-token\") pod \"e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.129173 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.129136 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:19.136546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.136518 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:19.173204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.173158 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s"] Oct 31 19:47:19.179325 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.179301 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222220 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222273 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222303 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222329 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsc8\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-kube-api-access-mhsc8\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222524 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222565 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222658 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222719 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.224073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.222749 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.232050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.230631 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s"] Oct 31 19:47:19.324081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.323967 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324155 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324189 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324215 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsc8\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-kube-api-access-mhsc8\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324362 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324380 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324403 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324561 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324442 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324561 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324507 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324666 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324569 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.324666 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.324601 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.325114 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.325083 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.325247 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.325138 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.327593 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.325653 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.327593 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.327374 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.327907 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.327886 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.379131 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.379094 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.379294 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.379181 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsc8\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-kube-api-access-mhsc8\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-tm95s\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.456150 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.456116 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh"] Oct 31 19:47:19.461530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.461510 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.469776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.469747 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj"] Oct 31 19:47:19.474534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.474510 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.475061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.475035 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" event={"ID":"78764e47-af66-4b6d-aa36-c47a927b2d3a","Type":"ContainerStarted","Data":"d85db04ccf435f0f0651f2bf379c16209072b032baad407ee9a3af696fc05552"} Oct 31 19:47:19.475154 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.475073 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" event={"ID":"78764e47-af66-4b6d-aa36-c47a927b2d3a","Type":"ContainerStarted","Data":"7d1975db037e7ccb4e04160ce6f91248ffcb3d43e9ba06ce316eb6f866a290fe"} Oct 31 19:47:19.498217 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.498193 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:19.514405 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.514371 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh"] Oct 31 19:47:19.528753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.527735 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.528753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.527786 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23229f4b-ec12-465d-ac48-1e7be99a337d-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.528753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.527825 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.528753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.527871 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.528753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.527906 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.528753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.527953 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.528753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.528037 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.529191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529017 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.529191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529067 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.529191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529102 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.529191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529145 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.529401 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529192 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84qkd\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-kube-api-access-84qkd\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.529401 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529251 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.529401 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529285 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.529401 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529337 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.529401 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529375 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bb78\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-kube-api-access-8bb78\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.529640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529444 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.529640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.529514 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.531517 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.531492 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd"] Oct 31 19:47:19.537777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.537753 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj"] Oct 31 19:47:19.541490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.541385 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.541490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.541469 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.541646 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.541533 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.548055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.547287 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:19.571003 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.570818 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.571003 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.570904 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.571302 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.570947 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.576411 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.576353 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk"] Oct 31 19:47:19.631019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.630971 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.631157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631053 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.631157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631115 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631152 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.631326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631189 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631234 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631284 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84qkd\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-kube-api-access-84qkd\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.631467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631348 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631381 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.631467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631438 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631476 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bb78\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-kube-api-access-8bb78\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631542 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631593 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631677 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.631743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631723 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23229f4b-ec12-465d-ac48-1e7be99a337d-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.632036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631766 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.632036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631802 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.632036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.631834 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.633190 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.633163 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.633452 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.633428 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.633723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.633699 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.634052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.634031 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-certs\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.634309 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.634286 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.634588 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.634567 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-credential-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.635207 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.635179 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-data\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.635655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.635626 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23229f4b-ec12-465d-ac48-1e7be99a337d-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.645373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.638118 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-socket\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.645373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.639407 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.645373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.645271 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.645373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.645291 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istiod-ca-cert\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.650206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.650174 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-podinfo\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.654733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.653742 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-envoy\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.694980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.694897 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84qkd\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-kube-api-access-84qkd\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.696494 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.696459 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.701350 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.699406 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bb78\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-kube-api-access-8bb78\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.701350 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.699839 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-token\") pod \"e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.790298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.787809 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:19.790298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.788890 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:19.925956 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.925902 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podStartSLOduration=1.9258869079999998 podStartE2EDuration="1.925886908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:19.794670809 +0000 UTC m=+1811.034893944" watchObservedRunningTime="2026-10-31 19:47:19.925886908 +0000 UTC m=+1811.166110036" Oct 31 19:47:19.926250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.926229 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s"] Oct 31 19:47:19.934527 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:47:19.933961 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c4c87e_9ef7_44a0_b61c_7939a1a40205.slice/crio-abac1fd65530abf199034447d25e94ece9adea07d606d24a7bda5084fcec0355 WatchSource:0}: Error finding container abac1fd65530abf199034447d25e94ece9adea07d606d24a7bda5084fcec0355: Status 404 returned error can't find the container with id abac1fd65530abf199034447d25e94ece9adea07d606d24a7bda5084fcec0355 Oct 31 19:47:19.937188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.936947 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.937188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.937047 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:19.937188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:19.937081 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:20.284441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.269929 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj"] Oct 31 19:47:20.314397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.313853 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:20.314397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.313941 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:20.314397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.313982 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:20.614329 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.614132 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh"] Oct 31 19:47:20.617050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.617009 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:20.618274 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.618238 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:20.618655 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:47:20.618167 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb6cda9_dd33_4a69_a1b2_8341a79026af.slice/crio-476325458acc792a486ec0e646d3c150d0e45eb6cb4474eabace97255ce911be WatchSource:0}: Error finding container 476325458acc792a486ec0e646d3c150d0e45eb6cb4474eabace97255ce911be: Status 404 returned error can't find the container with id 476325458acc792a486ec0e646d3c150d0e45eb6cb4474eabace97255ce911be Oct 31 19:47:20.625788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.625720 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:20.625909 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.625889 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:20.626026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.625985 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:20.630807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.630767 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" event={"ID":"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe","Type":"ContainerStarted","Data":"00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79"} Oct 31 19:47:20.630984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.630969 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" event={"ID":"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe","Type":"ContainerStarted","Data":"509649a422897025884d4950fc6d0976ff7930730473195c591ec71be51de88c"} Oct 31 19:47:20.640271 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.640177 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" event={"ID":"71c4c87e-9ef7-44a0-b61c-7939a1a40205","Type":"ContainerStarted","Data":"82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7"} Oct 31 19:47:20.640271 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.640232 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" event={"ID":"71c4c87e-9ef7-44a0-b61c-7939a1a40205","Type":"ContainerStarted","Data":"abac1fd65530abf199034447d25e94ece9adea07d606d24a7bda5084fcec0355"} Oct 31 19:47:20.648448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.648416 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" event={"ID":"23229f4b-ec12-465d-ac48-1e7be99a337d","Type":"ContainerStarted","Data":"9762ff7506e01670e293e74626bb34bf05b5e4beb376ccedba3d3f53239c1f31"} Oct 31 19:47:20.651797 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.651769 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" event={"ID":"a58dae4e-ef65-4a85-91ac-35bb3f5db867","Type":"ContainerStarted","Data":"62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553"} Oct 31 19:47:20.651926 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.651819 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" event={"ID":"a58dae4e-ef65-4a85-91ac-35bb3f5db867","Type":"ContainerStarted","Data":"f7e4dcf0f8af0647580c90ce60c96f7876fd1c98023aadc0a733429cf0e557ea"} Oct 31 19:47:20.914020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:20.901890 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podStartSLOduration=2.901855463 podStartE2EDuration="2.901855463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.7736312 +0000 UTC m=+1812.013854354" watchObservedRunningTime="2026-10-31 19:47:20.901855463 +0000 UTC m=+1812.142078593" Oct 31 19:47:21.112861 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.097884 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podStartSLOduration=3.09786628 podStartE2EDuration="3.09786628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:20.981181507 +0000 UTC m=+1812.221404631" watchObservedRunningTime="2026-10-31 19:47:21.09786628 +0000 UTC m=+1812.338089412" Oct 31 19:47:21.138361 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.138330 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:21.138787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.138766 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:21.143441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.143417 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.144572 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.144545 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:21.144898 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.143616 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.145068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.145048 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:21.506948 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.506906 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:21.509206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.509166 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.509431 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.509406 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:21.546508 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.546473 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.546825 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.546783 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:21.677486 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.677441 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" event={"ID":"2bb6cda9-dd33-4a69-a1b2-8341a79026af","Type":"ContainerStarted","Data":"d8cef71e5fd12d6a073b59309bf3f0d458070ffc2c65324c1d55704d51bb2898"} Oct 31 19:47:21.677754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.677727 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" event={"ID":"2bb6cda9-dd33-4a69-a1b2-8341a79026af","Type":"ContainerStarted","Data":"476325458acc792a486ec0e646d3c150d0e45eb6cb4474eabace97255ce911be"} Oct 31 19:47:21.707711 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.707671 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" event={"ID":"23229f4b-ec12-465d-ac48-1e7be99a337d","Type":"ContainerStarted","Data":"5196df3fdd4bc33f1edd8bd5b775673f5e192e39fa6d7e9d876a0267d430af3c"} Oct 31 19:47:21.802071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.798457 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:21.802071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.798527 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:21.802071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.798630 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.802071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.798670 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:21.802071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.798953 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:21.804195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.798980 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:21.836974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.836896 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podStartSLOduration=2.836871157 podStartE2EDuration="2.836871157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:21.097328053 +0000 UTC m=+1812.337551186" watchObservedRunningTime="2026-10-31 19:47:21.836871157 +0000 UTC m=+1813.077094287" Oct 31 19:47:21.838745 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.838685 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podStartSLOduration=2.83866926 podStartE2EDuration="2.83866926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:21.803980964 +0000 UTC m=+1813.044204095" watchObservedRunningTime="2026-10-31 19:47:21.83866926 +0000 UTC m=+1813.078892394" Oct 31 19:47:21.937021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:21.923164 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podStartSLOduration=2.923132526 podStartE2EDuration="2.923132526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:21.922269415 +0000 UTC m=+1813.162492549" watchObservedRunningTime="2026-10-31 19:47:21.923132526 +0000 UTC m=+1813.163355656" Oct 31 19:47:22.120350 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.120245 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:22.130477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.130441 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.130790 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.130744 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:22.137337 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.137308 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.137584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.137548 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:22.500024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.499895 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.500024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.499956 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:22.547448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.547407 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.548081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.547975 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:22.639143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.639082 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:22.639824 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.639412 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:22.818426 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.818312 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.818696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.818648 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:22.818908 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.818893 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:22.819054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.819037 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:22.987113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:22.987065 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.026597 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.026509 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:23.122296 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.122166 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.123043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.123019 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:47:23.129902 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.129875 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.130142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.130113 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:23.150025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.141804 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.150025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.141858 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:23.165030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.164348 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.660102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.652130 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:23.660102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.653518 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:23.660102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.653796 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:47:23.660102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.653850 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:47:23.665295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.660833 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.665295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.660878 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:23.665295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.661011 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.665295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.661040 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:23.872456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.867617 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.872456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.867681 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:23.872456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.869916 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:23.872456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.869971 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:23.973082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.953961 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.993018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.988229 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:23.993018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:23.988772 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:47:24.098018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.093476 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.098018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.093807 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:47:24.181619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.174933 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.181619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.175017 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:24.181619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.175126 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 19:47:24.181619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.175457 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.181619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.179310 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:47:24.181619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.180823 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.181619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.180882 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:24.299072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.288725 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 31 19:47:24.674092 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.610218 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:24.674092 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.610292 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:24.674092 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.646790 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.733016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.718224 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:24.809035 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.719297 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:24.946009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.945908 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.017050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.979497 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:25.080015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:24.978165 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:25.105866 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.105830 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.106171 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.106140 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:25.110017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.106360 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.246140 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.246090 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:25.246402 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.130229 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:25.246641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.107087 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 19:47:25.246813 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.132466 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.246946 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.246920 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:25.652069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.625806 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.652069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.625865 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:25.652069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.626583 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.652069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.626615 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:25.666068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.655057 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:47:25.724038 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.666390 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 19:47:25.724038 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.680278 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.724038 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.690629 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:25.876134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.876080 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:25.991513 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.991457 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 31 19:47:25.992508 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.992464 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.992739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.992711 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:25.993556 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.993529 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:25.993713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:25.993694 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:26.003116 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.003078 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q272n" podUID="6f8e9398-998f-42be-8dee-f0e169426697" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Oct 31 19:47:26.005076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.005042 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:26.013010 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.005555 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:26.024027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.010595 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn" podUID="7e7c0d83-8f3b-4130-9469-1d70e37eb9b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 19:47:26.113097 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:47:26.086044 6481 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.139s" Oct 31 19:47:26.113385 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.113367 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:47:26.196681 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.136214 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.196982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.196948 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:26.197193 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.137387 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.197325 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.197305 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:26.284269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.258805 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.301020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.263132 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.308275 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.308130 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.688543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.688439 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.873134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.731116 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:26.899729 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.763928 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.900045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.900012 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:26.900209 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.809078 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.900350 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.900329 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:26.900471 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.823103 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.900591 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.900575 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:26.900700 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.840236 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:26.900833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:26.900815 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:27.083019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.054463 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.111458 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.074083 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 31 19:47:27.133024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.111762 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:27.206467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.206423 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.207126 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.207096 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:27.221382 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.221270 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.225204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.225078 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:27.315671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.309184 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 19:47:27.315671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.310480 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:27.315671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.312695 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:27.518026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.517449 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.518026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.517513 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:27.552337 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.552302 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.552617 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.552587 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:27.799055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.798947 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.799329 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.799282 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:27.799818 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.799794 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:27.799984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.799962 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:27.924283 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:47:27.924236 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:47:27.935067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:27.933346 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.148442 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.147927 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.148842 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.148339 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.167033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.164083 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:28.167033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.149060 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.167033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.164173 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:28.173145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.173097 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.203134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.203082 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.343194 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.343149 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.344339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.344293 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.444185 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.311538 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:28.574064 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.511056 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.602198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.602140 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:28.768020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.756186 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.768436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.768382 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:28.842025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.833573 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.842333 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.842295 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:28.850018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.842523 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:28.850250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.850222 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:28.964110 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:28.957594 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.147075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.133913 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.147075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.133982 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:29.182019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.159104 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.182019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.160027 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.182019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.160079 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:29.182019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.176084 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.207165 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.207117 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.390462 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.390409 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.391521 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.391099 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:29.739020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.730026 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.763393 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.763352 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:29.763627 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.747939 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:29.764106 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.746660 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.764295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.764273 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:29.812125 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.807154 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.812125 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.807228 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:29.833046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.824627 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:29.833046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.824706 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:29.988024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:29.967988 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.159460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.159352 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.248018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.211586 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:30.248018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.215103 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.248018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.227062 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:30.248018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.215167 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:30.248018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.216232 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.248018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.227206 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:30.393043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.392072 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.393043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.392454 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:30.526030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.505674 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.526364 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.526322 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:30.615019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.590209 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.678018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.670068 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 19:47:30.771021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.770961 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:30.794575 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.794484 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.794892 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.794863 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:30.795073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.794806 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:30.795189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:30.795171 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:31.028019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.024599 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.129793 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.129709 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:31.130059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.130026 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:31.137836 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.137805 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:31.138077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.138051 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:31.235061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.230093 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.235061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.230451 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.302197 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.302158 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:47:31.452075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.416679 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:31.699480 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.488250 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.699959 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.570268 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:31.701120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.625424 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:31.701292 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.701262 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:31.806162 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.806110 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:31.876029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.862122 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:31.876356 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.876322 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:31.876500 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.863080 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:31.876620 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.876601 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:31.922119 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:31.916543 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.028204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.028160 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.151071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.148742 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.151071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.148821 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:32.151071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.149183 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.151071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.149238 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:32.232320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.232160 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.232729 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.232250 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.386019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.382880 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.513265 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.513231 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.513757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.513631 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:32.635175 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.635135 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:32.635511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.635460 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.697019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.692560 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:32.718061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.706081 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.726026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.706124 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:32.836723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.836601 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:32.850272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.850217 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.850580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.850531 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:32.850828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.850811 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:32.850970 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.850951 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:33.076791 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.076696 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.109718 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:32.714457 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.135331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.135299 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:33.135582 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.135557 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:33.144164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.144134 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:33.144397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.144372 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:33.253648 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.253600 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.254712 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.254680 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.383873 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.383778 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:47:33.551478 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.551444 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:33.686501 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.686404 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:33.686763 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.686728 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.722879 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.722843 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:33.771147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.771101 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:33.771431 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.756568 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.826662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.759771 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:33.826662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.756526 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:33.826662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.800215 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:33.826662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.821296 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:33.826662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.800267 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:33.826662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.821350 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:33.842186 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:33.842133 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:34.125654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.125605 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:34.133020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.129275 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:34.138150 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.138124 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:34.147752 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.147721 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 19:47:34.174021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.147951 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 19:47:34.174336 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.153129 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 19:47:34.174511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.174483 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 19:47:34.270114 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.270068 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:34.289059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.271536 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:34.289772 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.289366 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 31 19:47:34.492914 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.459587 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:34.521030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.514511 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:34.533399 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.533370 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:47:34.544195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.522321 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:34.545437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.545404 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:34.675334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.675285 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:34.825073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.800537 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:34.825073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.801830 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:34.975555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:34.975512 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.030987 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.059605 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.158930 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.132118 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.079957 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.159300 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.152202 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:35.173073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.159359 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:35.323280 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.300606 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:35.372341 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.348511 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:35.487719 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.487684 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:47:35.489312 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.489290 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:47:35.489530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.489514 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:47:35.496588 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.495231 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:47:35.500415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.500303 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 19:47:35.500415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.500358 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 19:47:35.503761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.503743 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:47:35.540343 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.540296 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:35.566796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.566760 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:47:35.747318 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.747243 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:47:35.814168 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.814132 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 19:47:35.814436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.814408 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 19:47:35.816538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.816516 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 19:47:35.816698 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:35.816674 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 19:47:36.158286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.158190 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:36.158517 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.158487 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:36.209920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.209891 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:36.228895 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.228091 6481 patch_prober.go:29] interesting pod/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:36.229260 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.229230 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:36.231608 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.231586 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:36.235301 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.235280 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:47:36.256235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.245967 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:47:36.845767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.842442 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:36.853930 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:36.853898 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:47:37.239287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:37.239250 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:37.240724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:37.240696 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:47:37.356822 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:37.356793 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:37.357767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:37.357747 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:47:48.229250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.229212 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr"] Oct 31 19:47:48.233762 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.233742 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.320814 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.320781 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr"] Oct 31 19:47:48.376668 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.376636 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.376846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.376679 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.376846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.376774 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.376846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.376803 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d382e91b-3398-4c58-9120-64b2e0f32ccf-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.376846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.376844 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.377025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.376949 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.377081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.377060 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.377161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.377147 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.377250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.377238 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p994t\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-kube-api-access-p994t\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.478678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.478644 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.478844 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.478691 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d382e91b-3398-4c58-9120-64b2e0f32ccf-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.478844 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.478736 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.478844 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.478759 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.478844 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.478816 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.478906 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.478976 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p994t\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-kube-api-access-p994t\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.479113 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.479156 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.479186 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.479249 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-data\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479566 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.479544 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-certs\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479566 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.479551 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-credential-socket\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.479678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.479579 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d382e91b-3398-4c58-9120-64b2e0f32ccf-istiod-ca-cert\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.481207 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.481182 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-envoy\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.481438 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.481421 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-podinfo\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.532149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.532116 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-token\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.532323 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.532300 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p994t\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-kube-api-access-p994t\") pod \"e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.547326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.547290 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:48.779926 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.779854 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght"] Oct 31 19:47:48.785302 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.785272 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.841032 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.840984 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght"] Oct 31 19:47:48.884744 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.884698 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.884953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.884834 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.884953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.884908 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.885133 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.885047 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.885133 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.885095 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.885241 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.885183 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.885304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.885271 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j244\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-kube-api-access-9j244\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.885542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.885403 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fed4fb-ffc4-42c8-a6b4-14c475306893-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.885542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.885501 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987313 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987278 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fed4fb-ffc4-42c8-a6b4-14c475306893-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987478 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987332 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987478 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987450 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987568 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987667 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987615 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987673 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987711 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987824 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987746 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.987824 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987776 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-credential-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.988270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.987930 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j244\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-kube-api-access-9j244\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.988270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.988110 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-data\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.988270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.988181 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-socket\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.988270 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.988181 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fed4fb-ffc4-42c8-a6b4-14c475306893-istiod-ca-cert\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.988778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.988756 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-certs\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.989650 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.989628 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr"] Oct 31 19:47:48.990530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.990513 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-envoy\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.990608 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.990586 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-podinfo\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:48.992355 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:47:48.992334 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd382e91b_3398_4c58_9120_64b2e0f32ccf.slice/crio-0f7d207e6ead784eac05b502523cdd09664430a795d98aed8e0e641078eedb1f WatchSource:0}: Error finding container 0f7d207e6ead784eac05b502523cdd09664430a795d98aed8e0e641078eedb1f: Status 404 returned error can't find the container with id 0f7d207e6ead784eac05b502523cdd09664430a795d98aed8e0e641078eedb1f Oct 31 19:47:48.994504 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.994477 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:48.994581 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.994551 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:48.994581 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:48.994577 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:49.040373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.040322 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-token\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:49.040467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.040400 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j244\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-kube-api-access-9j244\") pod \"e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:49.099987 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.099957 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:49.554008 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.551312 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght"] Oct 31 19:47:49.556060 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:47:49.556028 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fed4fb_ffc4_42c8_a6b4_14c475306893.slice/crio-aacff49a0dad3ae9af41c28c3089c87be8b6cfdb39543006ff888e030bc5dcff WatchSource:0}: Error finding container aacff49a0dad3ae9af41c28c3089c87be8b6cfdb39543006ff888e030bc5dcff: Status 404 returned error can't find the container with id aacff49a0dad3ae9af41c28c3089c87be8b6cfdb39543006ff888e030bc5dcff Oct 31 19:47:49.559025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.558961 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:49.559133 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.559049 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:49.559133 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.559099 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:49.934437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.934301 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" event={"ID":"d382e91b-3398-4c58-9120-64b2e0f32ccf","Type":"ContainerStarted","Data":"696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb"} Oct 31 19:47:49.934437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.934372 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" event={"ID":"d382e91b-3398-4c58-9120-64b2e0f32ccf","Type":"ContainerStarted","Data":"0f7d207e6ead784eac05b502523cdd09664430a795d98aed8e0e641078eedb1f"} Oct 31 19:47:49.939384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.939350 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" event={"ID":"41fed4fb-ffc4-42c8-a6b4-14c475306893","Type":"ContainerStarted","Data":"cc74997edef1f50b3044ee3192adde23f71e0ceb01777e28877bef8e927eaec0"} Oct 31 19:47:49.939498 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:49.939408 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" event={"ID":"41fed4fb-ffc4-42c8-a6b4-14c475306893","Type":"ContainerStarted","Data":"aacff49a0dad3ae9af41c28c3089c87be8b6cfdb39543006ff888e030bc5dcff"} Oct 31 19:47:50.040611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.040579 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7"] Oct 31 19:47:50.044920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.044897 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.054745 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.052731 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podStartSLOduration=2.052716663 podStartE2EDuration="2.052716663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:50.051674083 +0000 UTC m=+1841.291897213" watchObservedRunningTime="2026-10-31 19:47:50.052716663 +0000 UTC m=+1841.292939829" Oct 31 19:47:50.101239 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.101161 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:47:50.105214 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.105173 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:50.105384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.105251 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:50.105602 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.105579 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7"] Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112179 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112367 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112436 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4r7r\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-kube-api-access-d4r7r\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112560 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112608 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112653 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112683 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112732 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68856c11-a0fe-4119-9f1d-256ab6271cdb-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.118037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.112874 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214401 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214607 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214490 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214689 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214633 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214689 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214678 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4r7r\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-kube-api-access-d4r7r\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214812 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214781 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214860 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214847 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214881 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214954 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214911 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.214954 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.214947 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68856c11-a0fe-4119-9f1d-256ab6271cdb-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.215472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.215132 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-data\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.215472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.215245 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-credential-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.215472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.215353 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-socket\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.215472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.215410 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-certs\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.215779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.215738 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68856c11-a0fe-4119-9f1d-256ab6271cdb-istiod-ca-cert\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.217424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.217400 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-podinfo\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.219203 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.219185 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-envoy\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.294678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.293820 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4r7r\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-kube-api-access-d4r7r\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.295421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.295388 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-token\") pod \"e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.380019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.362273 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:50.547859 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.547826 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:47:50.549614 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.549550 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:50.549614 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.549595 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:50.835888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.835842 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podStartSLOduration=2.835826989 podStartE2EDuration="2.835826989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:50.294220366 +0000 UTC m=+1841.534443496" watchObservedRunningTime="2026-10-31 19:47:50.835826989 +0000 UTC m=+1842.076050116" Oct 31 19:47:50.837055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.837031 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7"] Oct 31 19:47:50.838229 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:47:50.838205 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68856c11_a0fe_4119_9f1d_256ab6271cdb.slice/crio-d121150ce11b6d45d66ce70be82ff7402af7fa7880482b2151d9a36f2ca1a0af WatchSource:0}: Error finding container d121150ce11b6d45d66ce70be82ff7402af7fa7880482b2151d9a36f2ca1a0af: Status 404 returned error can't find the container with id d121150ce11b6d45d66ce70be82ff7402af7fa7880482b2151d9a36f2ca1a0af Oct 31 19:47:50.841190 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.841159 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:50.841311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.841229 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:50.841311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.841275 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:47:50.947486 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:50.947442 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" event={"ID":"68856c11-a0fe-4119-9f1d-256ab6271cdb","Type":"ContainerStarted","Data":"d121150ce11b6d45d66ce70be82ff7402af7fa7880482b2151d9a36f2ca1a0af"} Oct 31 19:47:51.114084 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:51.109410 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:51.114084 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:51.109470 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:51.560651 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:51.559752 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:51.560651 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:51.559809 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:51.957433 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:51.957350 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" event={"ID":"68856c11-a0fe-4119-9f1d-256ab6271cdb","Type":"ContainerStarted","Data":"508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84"} Oct 31 19:47:52.109877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.102595 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:52.109877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.102674 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:52.200022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.185588 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podStartSLOduration=3.185563737 podStartE2EDuration="3.185563737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:52.126618989 +0000 UTC m=+1843.366842118" watchObservedRunningTime="2026-10-31 19:47:52.185563737 +0000 UTC m=+1843.425786867" Oct 31 19:47:52.362856 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.362825 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:47:52.363672 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.363652 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:52.363733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.363702 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:52.549022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.548098 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:52.549022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:52.548162 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:53.118019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:53.115530 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:53.118019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:53.115694 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:53.382060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:53.370978 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:53.382060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:53.371069 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:53.549425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:53.549392 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:53.549681 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:53.549650 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:54.103454 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.103415 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:54.103742 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.103714 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:54.247703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.247111 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.273840 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.268650 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.295177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.295078 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.395071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.382381 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:54.395071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.382439 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:54.620464 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.619898 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.621084 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.621052 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:54.621358 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.621333 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:54.719018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.716951 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:54.772490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.772433 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:54.835052 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:54.832139 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:47:55.106099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.105978 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:55.106354 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.106321 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:55.248372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.248325 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.250521 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.250493 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:47:55.265256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.265214 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.288093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.283808 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.362451 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.331397 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:47:55.380180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.380137 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:55.380478 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.380435 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:55.426217 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.426172 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.577544 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.577502 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:55.577808 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.577763 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:55.662077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.647643 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 31 19:47:55.778214 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.778171 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 19:47:55.778750 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.778726 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:47:55.802018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.785034 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.803467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.802540 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:47:55.851007 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.833008 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:55.908441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.905089 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:47:55.943238 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:55.943136 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.150415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.150375 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:56.172512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.172472 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:56.249237 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.249175 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:47:56.356201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.356145 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.434061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.414631 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.462033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.435031 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:47:56.462033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.436135 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:56.462033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.436179 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:56.462033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.436653 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.462033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.444535 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:47:56.583663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.579603 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:56.583663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.579659 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:56.780410 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.780365 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.812018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.806645 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:56.961078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:56.959337 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:57.152026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.126316 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:57.152026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.126398 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:57.213080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.155365 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:57.267149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.267110 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:47:57.414020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.389210 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:57.636078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.607405 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.071033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.066256 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:58.071033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.686836 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 19:47:58.142378 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.644709 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.226237 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.961690 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.346263 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.346152 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:47:58.736543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.848532 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.765727 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.686495 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.769123 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.769065 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:58.796043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.795328 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.796043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.795401 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:58.796043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.648202 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2" podUID="9de1e1d1-192a-488e-8f42-e013800a9dcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.796727 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.830542 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:47:58.796843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.796754 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:58.796843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:57.962309 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:47:58.805408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.462299 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.805668 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.805640 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:58.805803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.702915 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:58.805924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.805904 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:58.806155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:58.504135 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:47:59.140040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.134326 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:47:59.140040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.134394 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:47:59.140040 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:47:59.134656 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:47:59.182019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.174915 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 31 19:47:59.208568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.206668 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:47:59.234948 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.234352 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:47:59.511691 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.486393 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:47:59.577701 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.577674 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:47:59.589525 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.502818 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:47:59.589823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.589779 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:47:59.649487 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.649439 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:47:59.649768 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.649737 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:47:59.847342 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.847240 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 19:48:00.022731 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:47:59.968291 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:48:00.163072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.131847 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:00.204015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.203964 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:48:00.290020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.232209 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:48:00.290020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.267332 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:48:00.291618 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.291580 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:48:00.378720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.378687 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:48:00.379286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.379248 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:48:00.446288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.446168 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:48:00.623178 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.623136 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:48:00.783809 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.783766 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:48:00.784061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:00.740714 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:48:01.064066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.063297 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.126823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.105591 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:48:01.126823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.105648 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:48:01.213338 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.192876 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.245040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.215776 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.282655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.282575 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:01.401138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.394741 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:48:01.401138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.394806 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:48:01.534718 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.534637 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 31 19:48:01.544147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.543915 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:48:01.545522 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.545498 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:48:01.548249 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.548229 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 19:48:01.548349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:01.548268 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 19:48:02.104151 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:02.104114 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 19:48:02.104634 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:02.104237 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 19:48:02.363150 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:02.363051 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:48:02.363150 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:02.363099 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:48:02.699357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:02.699261 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:48:02.701752 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:02.701732 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:48:02.711453 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:02.711421 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:48:03.351823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.351789 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6"] Oct 31 19:48:03.352794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.352759 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" containerID="cri-o://af7d3ea68125aca848a35ba2476f8f47af50b8e657cffd9da1b169dd14c62c6a" gracePeriod=30 Oct 31 19:48:03.363654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.363611 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 19:48:03.363837 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.363800 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 19:48:03.372396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.372075 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd"] Oct 31 19:48:03.372396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.372301 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" containerID="cri-o://838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9" gracePeriod=30 Oct 31 19:48:03.380377 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.380349 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27"] Oct 31 19:48:03.380691 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.380615 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" containerID="cri-o://c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b" gracePeriod=30 Oct 31 19:48:03.398530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.398502 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd"] Oct 31 19:48:03.399050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.399022 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" containerID="cri-o://43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739" gracePeriod=30 Oct 31 19:48:03.654694 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.654592 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:48:03.655243 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.655213 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:48:03.656144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:03.656119 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 19:48:05.363553 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:05.363518 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:48:05.410283 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:05.407217 6481 patch_prober.go:29] interesting pod/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:05.410283 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:05.407287 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:05.793126 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:05.793088 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:48:05.974804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:05.974776 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:48:06.415017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:06.414965 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gvdt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:48:06.415517 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:06.415040 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:48:08.724511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.724468 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:48:08.793006 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.792971 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:48:08.811541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.811513 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:48:08.833438 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833407 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25d5609-fe3b-46eb-8f30-91211512a1f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d5609-fe3b-46eb-8f30-91211512a1f2-istiod-ca-cert\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.833560 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833455 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-certs\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.833642 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833585 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-token\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-token\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.833695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833639 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-podinfo\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.833695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833654 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-certs" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.833802 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833730 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-data\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.833802 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833758 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-kube-api-access-2fztn\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-kube-api-access-2fztn\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.833802 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833782 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25d5609-fe3b-46eb-8f30-91211512a1f2-istiod-ca-cert" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:08.833802 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833799 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-socket\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.834017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.833859 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-envoy\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.834144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.834122 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-credential-socket\") pod \"a25d5609-fe3b-46eb-8f30-91211512a1f2\" (UID: \"a25d5609-fe3b-46eb-8f30-91211512a1f2\") " Oct 31 19:48:08.834264 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.834167 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-socket" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.834474 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.834450 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-credential-socket" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.834931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.834904 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.835043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.834937 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.835043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.834954 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25d5609-fe3b-46eb-8f30-91211512a1f2-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.835043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.834969 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.835250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.835155 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-data" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.836137 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.836111 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-token" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:08.836247 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.836229 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-podinfo" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:48:08.836346 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.836327 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-kube-api-access-2fztn" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "kube-api-access-2fztn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:08.836434 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.836413 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-envoy" pod "a25d5609-fe3b-46eb-8f30-91211512a1f2" (UID: "a25d5609-fe3b-46eb-8f30-91211512a1f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.935915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.935832 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-kube-api-access-wgzvj\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-kube-api-access-wgzvj\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.935915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.935888 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-podinfo\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.936331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936304 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-certs\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.936416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936360 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-token\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-token\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.936416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936410 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19954c3d-e292-4748-bf7b-16f3cb235fd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19954c3d-e292-4748-bf7b-16f3cb235fd0-istiod-ca-cert\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.936529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936471 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857f1cc5-3044-4453-8c23-4df1fc80b673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1cc5-3044-4453-8c23-4df1fc80b673-istiod-ca-cert\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.936587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936559 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-data\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.936641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936574 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-certs" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.936641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936628 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-socket\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.936743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936666 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-token\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-token\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.936743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936728 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-socket\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.936842 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936765 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-certs\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.936894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936859 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-podinfo\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.936894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936880 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19954c3d-e292-4748-bf7b-16f3cb235fd0-istiod-ca-cert" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:08.937014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.936938 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-data\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.937077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.937041 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-envoy\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.937142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.937118 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-credential-socket\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.937227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.937208 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-credential-socket\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.937300 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.937285 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-envoy\") pod \"19954c3d-e292-4748-bf7b-16f3cb235fd0\" (UID: \"19954c3d-e292-4748-bf7b-16f3cb235fd0\") " Oct 31 19:48:08.937354 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.937344 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-kube-api-access-qsd5z\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-kube-api-access-qsd5z\") pod \"857f1cc5-3044-4453-8c23-4df1fc80b673\" (UID: \"857f1cc5-3044-4453-8c23-4df1fc80b673\") " Oct 31 19:48:08.938367 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938343 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.938470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938372 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.938470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938387 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.938470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938402 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fztn\" (UniqueName: \"kubernetes.io/projected/a25d5609-fe3b-46eb-8f30-91211512a1f2-kube-api-access-2fztn\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.938470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938417 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25d5609-fe3b-46eb-8f30-91211512a1f2-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.938470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938451 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.938470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938466 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19954c3d-e292-4748-bf7b-16f3cb235fd0-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:08.939979 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.938980 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-data" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.939979 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.939758 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857f1cc5-3044-4453-8c23-4df1fc80b673-istiod-ca-cert" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:08.940402 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.940225 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-socket" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.940402 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.940265 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-podinfo" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:48:08.940601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.940516 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-certs" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.940601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.940571 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-credential-socket" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.940601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.940584 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-data" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.940774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.940725 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-credential-socket" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.940952 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.940923 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-kube-api-access-wgzvj" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "kube-api-access-wgzvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:08.941308 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.941288 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-socket" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.941540 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.941492 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-token" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:08.942918 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.942898 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-envoy" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.944045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.944017 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-kube-api-access-qsd5z" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "kube-api-access-qsd5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:08.944516 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.944486 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-envoy" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:08.944598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.944507 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-podinfo" pod "19954c3d-e292-4748-bf7b-16f3cb235fd0" (UID: "19954c3d-e292-4748-bf7b-16f3cb235fd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:48:08.947407 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:08.947384 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-token" pod "857f1cc5-3044-4453-8c23-4df1fc80b673" (UID: "857f1cc5-3044-4453-8c23-4df1fc80b673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:09.017789 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.017748 6481 generic.go:356] "Generic (PLEG): container finished" podID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerID="43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739" exitCode=0 Oct 31 19:48:09.017963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.017812 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" event={"ID":"857f1cc5-3044-4453-8c23-4df1fc80b673","Type":"ContainerDied","Data":"43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739"} Oct 31 19:48:09.017963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.017896 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" event={"ID":"857f1cc5-3044-4453-8c23-4df1fc80b673","Type":"ContainerDied","Data":"6646f4a1277d21d37361ea0d9121444859d85bff2ff8f9ce42f3d84257e19f06"} Oct 31 19:48:09.017963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.017933 6481 scope.go:122] "RemoveContainer" containerID="43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739" Oct 31 19:48:09.019648 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.019077 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd" Oct 31 19:48:09.026855 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.026830 6481 generic.go:356] "Generic (PLEG): container finished" podID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerID="af7d3ea68125aca848a35ba2476f8f47af50b8e657cffd9da1b169dd14c62c6a" exitCode=0 Oct 31 19:48:09.027059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.027038 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" event={"ID":"d7838e08-5257-4a6c-8861-4331b2b2fd98","Type":"ContainerDied","Data":"af7d3ea68125aca848a35ba2476f8f47af50b8e657cffd9da1b169dd14c62c6a"} Oct 31 19:48:09.029217 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.029180 6481 generic.go:356] "Generic (PLEG): container finished" podID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerID="c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b" exitCode=0 Oct 31 19:48:09.029330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.029269 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" event={"ID":"19954c3d-e292-4748-bf7b-16f3cb235fd0","Type":"ContainerDied","Data":"c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b"} Oct 31 19:48:09.029330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.029309 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" Oct 31 19:48:09.029436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.029312 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27" event={"ID":"19954c3d-e292-4748-bf7b-16f3cb235fd0","Type":"ContainerDied","Data":"be69c0f7c11a9ac6cc878451ee874ceb63d037e98555bce3526b7a2942df850f"} Oct 31 19:48:09.029785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.029765 6481 scope.go:122] "RemoveContainer" containerID="43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739" Oct 31 19:48:09.030358 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:09.030333 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739\": container with ID starting with 43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739 not found: ID does not exist" containerID="43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739" Oct 31 19:48:09.030461 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.030369 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739"} err="failed to get container status \"43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739\": rpc error: code = NotFound desc = could not find container \"43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739\": container with ID starting with 43838470d207dccdf549358513959528cc094509faf6372c60b83ec68f6b1739 not found: ID does not exist" Oct 31 19:48:09.030461 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.030391 6481 scope.go:122] "RemoveContainer" containerID="c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b" Oct 31 19:48:09.031834 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.031813 6481 generic.go:356] "Generic (PLEG): container finished" podID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerID="838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9" exitCode=0 Oct 31 19:48:09.032003 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.031843 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" event={"ID":"a25d5609-fe3b-46eb-8f30-91211512a1f2","Type":"ContainerDied","Data":"838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9"} Oct 31 19:48:09.032113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.032050 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" event={"ID":"a25d5609-fe3b-46eb-8f30-91211512a1f2","Type":"ContainerDied","Data":"0412596d25632924b87385747a31255cdb0d5fbbd10eeac45bc826e7a086f667"} Oct 31 19:48:09.032113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.031878 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd" Oct 31 19:48:09.039804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039784 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039808 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857f1cc5-3044-4453-8c23-4df1fc80b673-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039820 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039829 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039839 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039847 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039856 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039867 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039875 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039882 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039891 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039899 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857f1cc5-3044-4453-8c23-4df1fc80b673-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.039903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039910 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19954c3d-e292-4748-bf7b-16f3cb235fd0-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.040475 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039918 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsd5z\" (UniqueName: \"kubernetes.io/projected/857f1cc5-3044-4453-8c23-4df1fc80b673-kube-api-access-qsd5z\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.040475 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039927 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgzvj\" (UniqueName: \"kubernetes.io/projected/19954c3d-e292-4748-bf7b-16f3cb235fd0-kube-api-access-wgzvj\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.040475 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.039935 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857f1cc5-3044-4453-8c23-4df1fc80b673-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.043373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.043350 6481 scope.go:122] "RemoveContainer" containerID="c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b" Oct 31 19:48:09.043782 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:09.043753 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b\": container with ID starting with c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b not found: ID does not exist" containerID="c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b" Oct 31 19:48:09.043916 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.043888 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b"} err="failed to get container status \"c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b\": rpc error: code = NotFound desc = could not find container \"c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b\": container with ID starting with c02b1a85275cd3754347b2dbec5eec2743b8ce33097124a4e694580a00f4d64b not found: ID does not exist" Oct 31 19:48:09.044070 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.043919 6481 scope.go:122] "RemoveContainer" containerID="838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9" Oct 31 19:48:09.060508 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.060428 6481 scope.go:122] "RemoveContainer" containerID="838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9" Oct 31 19:48:09.060933 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:09.060898 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9\": container with ID starting with 838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9 not found: ID does not exist" containerID="838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9" Oct 31 19:48:09.061090 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.060939 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9"} err="failed to get container status \"838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9\": rpc error: code = NotFound desc = could not find container \"838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9\": container with ID starting with 838eaa18283b1defcd7fede340d9904c93a4adb0a466ee025ca315bb39ab93a9 not found: ID does not exist" Oct 31 19:48:09.065058 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.065036 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:48:09.121400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.121372 6481 scope.go:122] "RemoveContainer" containerID="af7d3ea68125aca848a35ba2476f8f47af50b8e657cffd9da1b169dd14c62c6a" Oct 31 19:48:09.141023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.140978 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-socket\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141137 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-token\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-token\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141200 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-socket" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:09.141375 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141358 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-envoy\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141432 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141405 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7838e08-5257-4a6c-8861-4331b2b2fd98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7838e08-5257-4a6c-8861-4331b2b2fd98-istiod-ca-cert\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141449 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-credential-socket\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141535 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141496 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-data\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141535 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141526 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-podinfo\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141579 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-kube-api-access-ggb5b\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-kube-api-access-ggb5b\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141744 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141726 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-certs\") pod \"d7838e08-5257-4a6c-8861-4331b2b2fd98\" (UID: \"d7838e08-5257-4a6c-8861-4331b2b2fd98\") " Oct 31 19:48:09.141796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141765 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-credential-socket" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:09.141796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.141788 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-data" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:09.142120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.142099 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-certs" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:09.142481 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.142461 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.142601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.142586 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.142601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.142606 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.142711 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.142620 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.142711 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.142500 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7838e08-5257-4a6c-8861-4331b2b2fd98-istiod-ca-cert" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:09.143578 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.143555 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-envoy" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:09.143706 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.143687 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-token" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:09.143960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.143943 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-kube-api-access-ggb5b" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "kube-api-access-ggb5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:09.144176 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.144157 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-podinfo" pod "d7838e08-5257-4a6c-8861-4331b2b2fd98" (UID: "d7838e08-5257-4a6c-8861-4331b2b2fd98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:48:09.243847 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.243818 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.243847 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.243846 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.244075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.243856 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7838e08-5257-4a6c-8861-4331b2b2fd98-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.244075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.243867 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7838e08-5257-4a6c-8861-4331b2b2fd98-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.244075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.243877 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggb5b\" (UniqueName: \"kubernetes.io/projected/d7838e08-5257-4a6c-8861-4331b2b2fd98-kube-api-access-ggb5b\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:09.262894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.262860 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27"] Oct 31 19:48:09.276764 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.276726 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xkt27"] Oct 31 19:48:09.546204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.546134 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd"] Oct 31 19:48:09.548962 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.548938 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6x5qd"] Oct 31 19:48:09.647556 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.647525 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd"] Oct 31 19:48:09.652476 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:09.652446 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-czjbd"] Oct 31 19:48:10.038768 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.038726 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" event={"ID":"d7838e08-5257-4a6c-8861-4331b2b2fd98","Type":"ContainerDied","Data":"a7842fbc010b205b701f43bf8d988cf45aceb83799ed1cdfc126446ccaeabcf5"} Oct 31 19:48:10.039168 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.038771 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6" Oct 31 19:48:10.253948 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.253916 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6"] Oct 31 19:48:10.256644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.256620 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gvdt6"] Oct 31 19:48:10.881860 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.881823 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" path="/var/lib/kubelet/pods/19954c3d-e292-4748-bf7b-16f3cb235fd0/volumes" Oct 31 19:48:10.882378 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.882363 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" path="/var/lib/kubelet/pods/857f1cc5-3044-4453-8c23-4df1fc80b673/volumes" Oct 31 19:48:10.882810 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.882797 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" path="/var/lib/kubelet/pods/a25d5609-fe3b-46eb-8f30-91211512a1f2/volumes" Oct 31 19:48:10.883594 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:10.883580 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" path="/var/lib/kubelet/pods/d7838e08-5257-4a6c-8861-4331b2b2fd98/volumes" Oct 31 19:48:18.728186 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728143 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql"] Oct 31 19:48:18.728791 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728773 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" Oct 31 19:48:18.728831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728792 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" Oct 31 19:48:18.728831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728807 6481 cpu_manager.go:414] "containerMap: removing container" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" Oct 31 19:48:18.728831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728812 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" Oct 31 19:48:18.728831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728824 6481 cpu_manager.go:414] "containerMap: removing container" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" Oct 31 19:48:18.728831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728829 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" Oct 31 19:48:18.728980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728838 6481 cpu_manager.go:414] "containerMap: removing container" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" Oct 31 19:48:18.728980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728843 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" Oct 31 19:48:18.728980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728951 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7838e08-5257-4a6c-8861-4331b2b2fd98" containerName="istio-proxy" Oct 31 19:48:18.728980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728959 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="19954c3d-e292-4748-bf7b-16f3cb235fd0" containerName="istio-proxy" Oct 31 19:48:18.728980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728967 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25d5609-fe3b-46eb-8f30-91211512a1f2" containerName="istio-proxy" Oct 31 19:48:18.728980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.728973 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="857f1cc5-3044-4453-8c23-4df1fc80b673" containerName="istio-proxy" Oct 31 19:48:18.733949 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.733932 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788378 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788350 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788412 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6976e490-6df0-42c7-8812-c0950f08d982-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788498 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788580 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788612 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbxh\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-kube-api-access-5sbxh\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788731 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6976e490-6df0-42c7-8812-c0950f08d982-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788857 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788833 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788868 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.788966 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.788919 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.804926 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.804895 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql"] Oct 31 19:48:18.890263 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890229 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6976e490-6df0-42c7-8812-c0950f08d982-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890428 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890301 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890428 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890319 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890428 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890364 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890430 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890500 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6976e490-6df0-42c7-8812-c0950f08d982-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890543 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890616 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890651 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbxh\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-kube-api-access-5sbxh\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890705 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.890831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.890804 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-credential-socket\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.891061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.891033 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-data\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.891213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.891092 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6976e490-6df0-42c7-8812-c0950f08d982-istiod-ca-cert\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.891213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.891089 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-certs\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.892714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.892691 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-envoy\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.893041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.893021 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6976e490-6df0-42c7-8812-c0950f08d982-istio-podinfo\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.943348 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.943314 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-istio-token\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:18.943528 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:18.943432 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbxh\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-kube-api-access-5sbxh\") pod \"e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:19.047665 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.047593 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:19.467475 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.467442 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql"] Oct 31 19:48:19.507710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.507664 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:19.507955 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.507931 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:19.508113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.508041 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:19.682844 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.682807 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr"] Oct 31 19:48:19.687173 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.687154 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.701468 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.701447 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb"] Oct 31 19:48:19.705496 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.705478 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.739931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.739836 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr"] Oct 31 19:48:19.807017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.789935 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb"] Oct 31 19:48:19.807017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.806939 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807008 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.807316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807053 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.807316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807102 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx25\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-kube-api-access-gxx25\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807156 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807187 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.807316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807220 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807255 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807316 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807305 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807348 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.807669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807374 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmwbg\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-kube-api-access-bmwbg\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.807669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807446 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807491 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e04162-c773-4620-90c1-f8eb6f659d1f-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.807669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807543 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9733ccf-04a4-4b33-913e-143456ef98a0-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.807669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807570 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.807669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807632 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.808028 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807689 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.808028 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.807726 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.908858 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.908820 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9733ccf-04a4-4b33-913e-143456ef98a0-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.908858 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.908854 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.908983 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909063 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909134 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909091 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909154 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909181 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909220 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909266 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909268 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx25\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-kube-api-access-gxx25\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909359 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909422 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909473 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909514 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909570 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909615 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909622 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9733ccf-04a4-4b33-913e-143456ef98a0-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909641 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwbg\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-kube-api-access-bmwbg\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909650 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909776 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909782 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909835 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909846 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-certs\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909882 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e04162-c773-4620-90c1-f8eb6f659d1f-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.909931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.909893 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.910584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.910170 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-data\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.910584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.910251 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-credential-socket\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.910710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.910660 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e04162-c773-4620-90c1-f8eb6f659d1f-istiod-ca-cert\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.912107 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.912079 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.912492 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.912467 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-envoy\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.912578 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.912467 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.912708 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.912691 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-podinfo\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.962979 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.962945 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.963174 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.963075 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx25\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-kube-api-access-gxx25\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:19.963714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.963683 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-token\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:19.964321 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:19.964299 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwbg\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-kube-api-access-bmwbg\") pod \"e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:20.001097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.000982 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:20.019606 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.019574 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:20.121330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.120941 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" event={"ID":"6976e490-6df0-42c7-8812-c0950f08d982","Type":"ContainerStarted","Data":"385cb29c6dbf9464451b17213ea10481418e324893a2aed66b1335bcab0f4f8f"} Oct 31 19:48:20.121330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.121051 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" event={"ID":"6976e490-6df0-42c7-8812-c0950f08d982","Type":"ContainerStarted","Data":"d41890ecee228bbc252ba7f64525e247015562026a8fe915d5257ea6e869c80b"} Oct 31 19:48:20.236235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.236186 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podStartSLOduration=2.236170897 podStartE2EDuration="2.236170897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:20.232732525 +0000 UTC m=+1871.472955655" watchObservedRunningTime="2026-10-31 19:48:20.236170897 +0000 UTC m=+1871.476394023" Oct 31 19:48:20.493732 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.493705 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb"] Oct 31 19:48:20.497013 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:48:20.496968 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9733ccf_04a4_4b33_913e_143456ef98a0.slice/crio-227f806f86fc656e3100acff2e69f3957101b763e3381aa1fb53100abb3bf999 WatchSource:0}: Error finding container 227f806f86fc656e3100acff2e69f3957101b763e3381aa1fb53100abb3bf999: Status 404 returned error can't find the container with id 227f806f86fc656e3100acff2e69f3957101b763e3381aa1fb53100abb3bf999 Oct 31 19:48:20.497204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.497180 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr"] Oct 31 19:48:20.497494 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:48:20.497471 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e04162_c773_4620_90c1_f8eb6f659d1f.slice/crio-6acea23f25df4ed1fe415f30e2c40c7d59813e7f322432509a4be157007ae361 WatchSource:0}: Error finding container 6acea23f25df4ed1fe415f30e2c40c7d59813e7f322432509a4be157007ae361: Status 404 returned error can't find the container with id 6acea23f25df4ed1fe415f30e2c40c7d59813e7f322432509a4be157007ae361 Oct 31 19:48:20.500177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.500144 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:20.500273 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.500216 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:20.500273 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.500256 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:20.501461 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.501433 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:20.501520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.501492 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:20.501557 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:20.501525 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:21.056059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.055204 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:48:21.056059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.055266 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 19:48:21.056059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.055564 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:21.171928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.171890 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" event={"ID":"c8e04162-c773-4620-90c1-f8eb6f659d1f","Type":"ContainerStarted","Data":"f44888c99d5e8ac8b69dc3e57a5cf3329c7da96885a22b826edacb148d252feb"} Oct 31 19:48:21.173857 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.173806 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" event={"ID":"c8e04162-c773-4620-90c1-f8eb6f659d1f","Type":"ContainerStarted","Data":"6acea23f25df4ed1fe415f30e2c40c7d59813e7f322432509a4be157007ae361"} Oct 31 19:48:21.203019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.184812 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" event={"ID":"f9733ccf-04a4-4b33-913e-143456ef98a0","Type":"ContainerStarted","Data":"16b7e76caa73ed9765c8e3d7a6dafa8bbc02d80cc5dd43189957d402e5315188"} Oct 31 19:48:21.203210 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.203089 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" event={"ID":"f9733ccf-04a4-4b33-913e-143456ef98a0","Type":"ContainerStarted","Data":"227f806f86fc656e3100acff2e69f3957101b763e3381aa1fb53100abb3bf999"} Oct 31 19:48:21.296701 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.296498 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podStartSLOduration=2.296472059 podStartE2EDuration="2.296472059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:21.29266715 +0000 UTC m=+1872.532890282" watchObservedRunningTime="2026-10-31 19:48:21.296472059 +0000 UTC m=+1872.536695178" Oct 31 19:48:21.413766 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:21.413639 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podStartSLOduration=2.41362165 podStartE2EDuration="2.41362165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:21.411558217 +0000 UTC m=+1872.651781343" watchObservedRunningTime="2026-10-31 19:48:21.41362165 +0000 UTC m=+1872.653844781" Oct 31 19:48:22.011019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.010581 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:22.011240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.011110 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:48:22.011240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.011199 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 19:48:22.032019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.022885 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:22.032019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.023024 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:48:22.032019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.023086 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 19:48:22.050511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.050468 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:48:22.050705 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:22.050544 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 19:48:23.004018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.003382 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:48:23.004018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.003447 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 19:48:23.035017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.029075 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:48:23.035017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.029131 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 19:48:23.048938 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.048898 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:48:23.049110 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.048952 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 19:48:23.347363 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.347262 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11068-a753a0"] Oct 31 19:48:23.353524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.353500 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:23.444268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.406842 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11068-a753a0"] Oct 31 19:48:23.444268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.409334 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xptzx\"" Oct 31 19:48:23.452399 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.450045 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9grn\" (UniqueName: \"kubernetes.io/projected/43e80bf6-f11a-42ef-95ea-d4648484f6c0-kube-api-access-n9grn\") pod \"test-curl-11068-a753a0\" (UID: \"43e80bf6-f11a-42ef-95ea-d4648484f6c0\") " pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:23.553394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.553358 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9grn\" (UniqueName: \"kubernetes.io/projected/43e80bf6-f11a-42ef-95ea-d4648484f6c0-kube-api-access-n9grn\") pod \"test-curl-11068-a753a0\" (UID: \"43e80bf6-f11a-42ef-95ea-d4648484f6c0\") " pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:23.654251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.654168 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9grn\" (UniqueName: \"kubernetes.io/projected/43e80bf6-f11a-42ef-95ea-d4648484f6c0-kube-api-access-n9grn\") pod \"test-curl-11068-a753a0\" (UID: \"43e80bf6-f11a-42ef-95ea-d4648484f6c0\") " pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:23.746066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:23.744258 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:24.019051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:24.001437 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:48:24.019051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:24.001497 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 19:48:24.049541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:24.048502 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:48:24.049541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:24.048562 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 19:48:24.049945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:24.049853 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:48:24.049945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:24.049892 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 19:48:24.302019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:24.298915 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11068-a753a0"] Oct 31 19:48:25.003848 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:25.002218 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:48:25.003848 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:25.002296 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 19:48:25.024787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:25.022064 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:48:25.024787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:25.022141 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 19:48:25.049749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:25.049473 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:48:25.049749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:25.049525 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 19:48:25.661175 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:25.661134 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11068-a753a0" event={"ID":"43e80bf6-f11a-42ef-95ea-d4648484f6c0","Type":"ContainerStarted","Data":"adb47147bbbd2479985e5fb10fb8c1c92d8fed7566795ec496a6516034de72a8"} Oct 31 19:48:26.001331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:26.001293 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:48:26.002102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:26.002070 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 19:48:26.024383 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:26.024350 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:48:26.024622 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:26.024597 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 19:48:26.048400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:26.048375 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:48:26.048816 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:26.048794 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 19:48:27.004837 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:27.004376 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:48:27.004837 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:27.004437 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 19:48:27.024512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:27.024385 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:48:27.024512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:27.024459 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 19:48:27.343425 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:27.343307 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:48:27.691533 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:27.689724 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:27.691533 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:27.691321 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:27.692699 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:27.692644 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:48:28.009547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:28.009511 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:48:28.009753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:28.009571 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 19:48:28.020462 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:28.020431 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 19:48:28.020600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:28.020491 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 19:48:29.033078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:29.033035 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:29.039015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:29.034311 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:29.042024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:29.041830 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:48:30.088232 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:30.087641 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:30.088232 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:30.087701 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.175019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.165116 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:31.175019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.165188 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.239058 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.238208 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.278183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.278139 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:48:31.442041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.440054 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:48:31.498098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.497432 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.784769 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.784713 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.815142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.815090 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:31.970576 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:31.970097 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.081209 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.081106 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.210200 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.210140 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.211928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.211476 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:32.211928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.211526 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.249039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.241103 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:32.249039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.241545 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:48:32.282121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.282077 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 19:48:32.293023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.282716 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:48:32.660064 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.635152 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:48:32.703017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.660887 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:48:32.819046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.799194 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 19:48:33.123018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:33.119444 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.159024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:32.820008 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.302256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:33.302211 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.304899 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:33.304830 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:33.458076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:33.415637 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.665219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:33.665073 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:33.916080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:33.915869 6481 patch_prober.go:29] interesting pod/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:33.944018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:33.942081 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.024431 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.024361 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.153233 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.153163 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.165062 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.161600 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 31 19:48:34.183050 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:34.162679 6481 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.235s" Oct 31 19:48:34.193055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.187072 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:48:34.195794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.195751 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:48:34.209262 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.209236 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:48:34.209551 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.209535 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:48:34.303038 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.302970 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:48:34.458094 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.449196 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:34.498018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.479537 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.510022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.504796 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.672029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.641254 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:34.744529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.744491 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:34.747504 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.747467 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:34.803038 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.802176 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:48:34.840935 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.840894 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 31 19:48:34.992122 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.992079 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:34.992311 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:34.992142 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.035775 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.035669 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:35.036073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.036040 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.036313 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.036288 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.153848 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.153805 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.170153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.170108 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.370245 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.370146 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.485362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.485308 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 31 19:48:35.504921 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.504865 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.641505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.641400 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.857276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.857231 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.894970 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.894890 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:35.895250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.895215 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:35.918119 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.918078 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 31 19:48:35.983912 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:35.979352 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 19:48:36.004139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:36.004105 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 19:48:36.127916 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:36.127533 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 19:48:36.288715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:36.288677 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:48:36.312018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:36.304038 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:36.506988 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:36.506939 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:36.728437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:36.710374 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 19:48:36.966051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:36.934367 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:37.358028 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:37.250015 6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:37.547538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:37.547016 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:37.588601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:37.588570 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 19:48:37.614287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:37.614205 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 19:48:38.502655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:38.502618 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11068-a753a0" event={"ID":"43e80bf6-f11a-42ef-95ea-d4648484f6c0","Type":"ContainerStarted","Data":"1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9"} Oct 31 19:48:38.609953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:38.609901 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11068-a753a0" podStartSLOduration=2.468783139 podStartE2EDuration="15.609885174s" totalImagesPullingTime="13.141102035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:38.607414619 +0000 UTC m=+1889.847637750" watchObservedRunningTime="2026-10-31 19:48:38.609885174 +0000 UTC m=+1889.850108307" Oct 31 19:48:40.091716 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.091685 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11068-a753a0"] Oct 31 19:48:40.098430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.098401 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11068-a753a0"] Oct 31 19:48:40.331179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.331144 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq"] Oct 31 19:48:40.331802 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.331788 6481 cpu_manager.go:414] "containerMap: removing container" podUID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" containerName="test-curl-11068-a753a0" Oct 31 19:48:40.331852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.331804 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" containerName="test-curl-11068-a753a0" Oct 31 19:48:40.331931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.331920 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" containerName="test-curl-11068-a753a0" Oct 31 19:48:40.335485 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.335466 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.406467 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.406432 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9be7bd13-openshift-default-dockercfg-bjfxc\"" Oct 31 19:48:40.630072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.600941 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-socket\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.630072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.601025 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-token\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.630072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.601066 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-credential-socket\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.630072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.601101 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-podinfo\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.630072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.597981 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq"] Oct 31 19:48:40.639531 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.638942 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-certs\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.639531 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.639284 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-data\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.639531 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.639405 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-envoy\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.642029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.640604 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkfx4\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-kube-api-access-tkfx4\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.642029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.640665 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istiod-ca-cert\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.689122 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.689077 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11068-a753a0" podUID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" containerName="test-curl-11068-a753a0" containerID="cri-o://1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9" gracePeriod=2 Oct 31 19:48:40.736963 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:40.736913 6481 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11068-a753a0\" is forbidden: User \"system:node:ip-10-0-143-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-west-2.compute.internal' and this object" podUID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:40.745252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.745198 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkfx4\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-kube-api-access-tkfx4\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.745522 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.745502 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istiod-ca-cert\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.745703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.745675 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-socket\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.745859 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.745819 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-token\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.746013 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.745985 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-credential-socket\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.746136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.746123 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-podinfo\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.746330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.746316 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-certs\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.746526 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.746483 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-data\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.746766 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.746736 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-envoy\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.752415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.752389 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istiod-ca-cert\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.773801 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.752909 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-socket\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.776788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.776757 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-certs\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.777335 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.777305 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-credential-socket\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.777726 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.777656 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-data\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.831465 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.831423 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-envoy\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.837041 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.835126 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkfx4\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-kube-api-access-tkfx4\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.839504 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.839446 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-token\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.840071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.840038 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-podinfo\") pod \"e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:40.929227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:40.929138 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:41.391397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.391373 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq"] Oct 31 19:48:41.392349 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:48:41.392328 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f56e5ba_5936_4ca9_b862_2064e8bb1f09.slice/crio-808ed7f3f7c8ecd94e2c9225c9084df86f047c55c44558e39ef52a5109a084ee WatchSource:0}: Error finding container 808ed7f3f7c8ecd94e2c9225c9084df86f047c55c44558e39ef52a5109a084ee: Status 404 returned error can't find the container with id 808ed7f3f7c8ecd94e2c9225c9084df86f047c55c44558e39ef52a5109a084ee Oct 31 19:48:41.398362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.398327 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:41.398451 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.398395 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:41.398451 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.398428 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:41.697681 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.697600 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" event={"ID":"2f56e5ba-5936-4ca9-b862-2064e8bb1f09","Type":"ContainerStarted","Data":"b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82"} Oct 31 19:48:41.697681 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.697653 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" event={"ID":"2f56e5ba-5936-4ca9-b862-2064e8bb1f09","Type":"ContainerStarted","Data":"808ed7f3f7c8ecd94e2c9225c9084df86f047c55c44558e39ef52a5109a084ee"} Oct 31 19:48:41.806103 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.806045 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" podStartSLOduration=1.80602645 podStartE2EDuration="1.80602645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:41.804902531 +0000 UTC m=+1893.045125673" watchObservedRunningTime="2026-10-31 19:48:41.80602645 +0000 UTC m=+1893.046249578" Oct 31 19:48:41.935159 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:41.935119 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:42.930471 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:42.930441 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:42.936059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:42.936035 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:43.005969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.005928 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e80bf6-f11a-42ef-95ea-d4648484f6c0-kube-api-access-n9grn\" (UniqueName: \"kubernetes.io/projected/43e80bf6-f11a-42ef-95ea-d4648484f6c0-kube-api-access-n9grn\") pod \"43e80bf6-f11a-42ef-95ea-d4648484f6c0\" (UID: \"43e80bf6-f11a-42ef-95ea-d4648484f6c0\") " Oct 31 19:48:43.008138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.008099 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e80bf6-f11a-42ef-95ea-d4648484f6c0-kube-api-access-n9grn" pod "43e80bf6-f11a-42ef-95ea-d4648484f6c0" (UID: "43e80bf6-f11a-42ef-95ea-d4648484f6c0"). InnerVolumeSpecName "kube-api-access-n9grn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:43.108872 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.108825 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9grn\" (UniqueName: \"kubernetes.io/projected/43e80bf6-f11a-42ef-95ea-d4648484f6c0-kube-api-access-n9grn\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:43.712873 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.712838 6481 generic.go:356] "Generic (PLEG): container finished" podID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" containerID="1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9" exitCode=137 Oct 31 19:48:43.713060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.712880 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:43.713060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.712934 6481 scope.go:122] "RemoveContainer" containerID="1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9" Oct 31 19:48:43.713280 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.713263 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:43.714434 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.714418 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:48:43.722862 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.722844 6481 scope.go:122] "RemoveContainer" containerID="1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9" Oct 31 19:48:43.723190 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:43.723169 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9\": container with ID starting with 1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9 not found: ID does not exist" containerID="1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9" Oct 31 19:48:43.723255 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:43.723199 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9"} err="failed to get container status \"1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9\": rpc error: code = NotFound desc = could not find container \"1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9\": container with ID starting with 1175aa4acaf52cd0ea9f0f5f35a801abc943b16e3892f43c0dd4e247bfe5d1a9 not found: ID does not exist" Oct 31 19:48:43.916103 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:43.916049 6481 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11068-a753a0\" is forbidden: User \"system:node:ip-10-0-143-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-west-2.compute.internal' and this object" podUID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" pod="opendatahub/test-curl-11068-a753a0" Oct 31 19:48:44.895297 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:44.895236 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e80bf6-f11a-42ef-95ea-d4648484f6c0" path="/var/lib/kubelet/pods/43e80bf6-f11a-42ef-95ea-d4648484f6c0/volumes" Oct 31 19:48:46.189785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.189752 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8"] Oct 31 19:48:46.194339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.194319 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.244517 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.244480 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ae778663-openshift-default-dockercfg-mmnl8\"" Oct 31 19:48:46.284671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.284636 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-envoy\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.284850 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.284694 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-data\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.284850 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.284795 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-credential-socket\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.284973 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.284856 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-certs\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.284973 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.284953 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp84g\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-kube-api-access-sp84g\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.285097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.284979 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7724f7-e34e-4212-9805-6c09fc227c18-istiod-ca-cert\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.285097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.285078 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-socket\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.285205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.285153 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-podinfo\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.285260 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.285223 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-token\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.296149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.296117 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8"] Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402091 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-token\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402249 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-envoy\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402311 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-data\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402357 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-credential-socket\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402420 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-certs\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402480 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp84g\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-kube-api-access-sp84g\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402520 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7724f7-e34e-4212-9805-6c09fc227c18-istiod-ca-cert\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402584 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-socket\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.403045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.402730 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-podinfo\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.428430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.407701 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-credential-socket\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.428430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.409753 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-data\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.428430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.410648 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-certs\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.428430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.411325 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7724f7-e34e-4212-9805-6c09fc227c18-istiod-ca-cert\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.428430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.411713 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-socket\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.428430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.414186 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-envoy\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.497072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.475221 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-podinfo\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.497072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.480248 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-token\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.497072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.482371 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp84g\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-kube-api-access-sp84g\") pod \"e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:46.531084 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:46.516741 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:47.281694 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:47.281650 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8"] Oct 31 19:48:47.329018 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:48:47.322698 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7724f7_e34e_4212_9805_6c09fc227c18.slice/crio-c62e2817f7fd2abc5cfad6533321a0385de6ba15269a3316deb59fd3d1558aa7 WatchSource:0}: Error finding container c62e2817f7fd2abc5cfad6533321a0385de6ba15269a3316deb59fd3d1558aa7: Status 404 returned error can't find the container with id c62e2817f7fd2abc5cfad6533321a0385de6ba15269a3316deb59fd3d1558aa7 Oct 31 19:48:47.379068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:47.364347 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:47.379068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:47.364452 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:47.379068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:47.364505 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:48:47.996304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:47.996268 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" event={"ID":"9a7724f7-e34e-4212-9805-6c09fc227c18","Type":"ContainerStarted","Data":"b1f99b67b77b61dacb5fb619020081240687b469c9e45982a736241e0f5f68f7"} Oct 31 19:48:47.996572 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:47.996555 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" event={"ID":"9a7724f7-e34e-4212-9805-6c09fc227c18","Type":"ContainerStarted","Data":"c62e2817f7fd2abc5cfad6533321a0385de6ba15269a3316deb59fd3d1558aa7"} Oct 31 19:48:48.151866 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:48.151813 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podStartSLOduration=2.151799769 podStartE2EDuration="2.151799769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:48.107431598 +0000 UTC m=+1899.347654764" watchObservedRunningTime="2026-10-31 19:48:48.151799769 +0000 UTC m=+1899.392022898" Oct 31 19:48:48.524395 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:48.524362 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:48.529677 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:48.529653 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:49.003187 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:49.003156 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:49.004149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:48:49.004131 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:48:57.327390 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:48:57.327349 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:49:04.885040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:04.870930 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:49:04.885040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:04.871062 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:49:26.630550 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.630517 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv"] Oct 31 19:49:26.636570 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.636549 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.689760 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.689725 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv"] Oct 31 19:49:26.697703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.697667 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfr8\" (UniqueName: \"kubernetes.io/projected/df04dc87-6cba-4037-9724-c5d66903d7dd-kube-api-access-rbfr8\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.697886 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.697776 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.698007 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.697962 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df04dc87-6cba-4037-9724-c5d66903d7dd-openshift-service-ca\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.698297 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.698273 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-metrics-certs\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.730848 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.730812 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9be7bd13-serving-cert\"" Oct 31 19:49:26.800237 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.800203 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df04dc87-6cba-4037-9724-c5d66903d7dd-openshift-service-ca\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.800505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.800486 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-metrics-certs\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.800568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.800543 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfr8\" (UniqueName: \"kubernetes.io/projected/df04dc87-6cba-4037-9724-c5d66903d7dd-kube-api-access-rbfr8\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.800641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.800625 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.800830 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:49:26.800803 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-serving-cert: secret "maas-api-e2e-ait-9be7bd13-serving-cert" not found Oct 31 19:49:26.800987 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:49:26.800880 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls podName:df04dc87-6cba-4037-9724-c5d66903d7dd nodeName:}" failed. No retries permitted until 2026-10-31 19:49:27.300859762 +0000 UTC m=+1938.541082886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls") pod "maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" (UID: "df04dc87-6cba-4037-9724-c5d66903d7dd") : secret "maas-api-e2e-ait-9be7bd13-serving-cert" not found Oct 31 19:49:26.800987 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.800925 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df04dc87-6cba-4037-9724-c5d66903d7dd-openshift-service-ca\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.803086 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.803063 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-metrics-certs\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:26.854198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:26.854170 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfr8\" (UniqueName: \"kubernetes.io/projected/df04dc87-6cba-4037-9724-c5d66903d7dd-kube-api-access-rbfr8\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:27.327185 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:49:27.327151 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:49:27.331161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:27.331138 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:27.333413 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:27.333396 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls\") pod \"maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:27.550278 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:27.550248 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:27.920645 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:27.920614 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv"] Oct 31 19:49:27.921918 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:49:27.921892 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf04dc87_6cba_4037_9724_c5d66903d7dd.slice/crio-9c025bf71eae1d012a0fc4d084f1142c5ca065f7a67b54cf2c7077cd21119554 WatchSource:0}: Error finding container 9c025bf71eae1d012a0fc4d084f1142c5ca065f7a67b54cf2c7077cd21119554: Status 404 returned error can't find the container with id 9c025bf71eae1d012a0fc4d084f1142c5ca065f7a67b54cf2c7077cd21119554 Oct 31 19:49:28.424276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:28.424238 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" event={"ID":"df04dc87-6cba-4037-9724-c5d66903d7dd","Type":"ContainerStarted","Data":"9c025bf71eae1d012a0fc4d084f1142c5ca065f7a67b54cf2c7077cd21119554"} Oct 31 19:49:29.432262 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:29.432220 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" event={"ID":"df04dc87-6cba-4037-9724-c5d66903d7dd","Type":"ContainerStarted","Data":"cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf"} Oct 31 19:49:29.432742 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:29.432304 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:29.542455 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:29.542399 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" podStartSLOduration=2.6640002259999997 podStartE2EDuration="3.542376975s" totalImagesPullingTime="878.376749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:29.538839366 +0000 UTC m=+1940.779062495" watchObservedRunningTime="2026-10-31 19:49:29.542376975 +0000 UTC m=+1940.782600108" Oct 31 19:49:34.123685 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.123653 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f"] Oct 31 19:49:34.128008 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.127974 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.169269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.169235 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4"] Oct 31 19:49:34.173725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.173700 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.177040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.177012 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5cqm6\"" Oct 31 19:49:34.221027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.220975 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f"] Oct 31 19:49:34.269699 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.269666 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4"] Oct 31 19:49:34.297696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.297661 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.297894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.297715 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.297894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.297830 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.297894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.297886 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.298126 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298058 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltrk\" (UniqueName: \"kubernetes.io/projected/fe5137bc-2028-4ac9-bc02-354efba8d848-kube-api-access-4ltrk\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.298126 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298097 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.298238 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298126 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5137bc-2028-4ac9-bc02-354efba8d848-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.298293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298222 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.298293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298269 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.298396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298305 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.298396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298375 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.298503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298394 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4cn\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-kube-api-access-mq4cn\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.298503 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298425 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.298608 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298491 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.298671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298602 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.298671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298631 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.298671 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298663 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.298835 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.298722 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.400597 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400502 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5137bc-2028-4ac9-bc02-354efba8d848-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.400597 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400581 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.400597 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400603 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.400902 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400621 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.400902 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400656 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.400902 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400674 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4cn\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-kube-api-access-mq4cn\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.400902 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400896 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.400942 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401025 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401051 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401084 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401092 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401114 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401132 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.401524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401216 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401288 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.401524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401349 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.401524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401383 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401383 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe5137bc-2028-4ac9-bc02-354efba8d848-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401405 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401472 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401586 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltrk\" (UniqueName: \"kubernetes.io/projected/fe5137bc-2028-4ac9-bc02-354efba8d848-kube-api-access-4ltrk\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.401897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401622 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.401897 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401640 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.402079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401917 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.402079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.401977 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.402079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.402021 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.402229 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.402155 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.403379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.403352 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.403611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.403588 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.403862 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.403843 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.404159 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.404142 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.454048 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.453977 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe5137bc-2028-4ac9-bc02-354efba8d848-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.454048 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.454041 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.454287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.454211 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltrk\" (UniqueName: \"kubernetes.io/projected/fe5137bc-2028-4ac9-bc02-354efba8d848-kube-api-access-4ltrk\") pod \"maas-default-gateway-openshift-default-58d898bd55-rxs7f\" (UID: \"fe5137bc-2028-4ac9-bc02-354efba8d848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.454373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.454352 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4cn\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-kube-api-access-mq4cn\") pod \"maas-default-gateway-openshift-default-58d898bd55-8q6r4\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.487710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.487677 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:34.741616 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.741582 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:34.908219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.908189 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4"] Oct 31 19:49:34.908708 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:49:34.908675 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfe4b38_38bc_4278_b1b2_3cf3e41b8c25.slice/crio-c7eee9d8bffcbf707d0d9477834bfff87d024a5a851ce1a8fde75e9a8bdcbbb0 WatchSource:0}: Error finding container c7eee9d8bffcbf707d0d9477834bfff87d024a5a851ce1a8fde75e9a8bdcbbb0: Status 404 returned error can't find the container with id c7eee9d8bffcbf707d0d9477834bfff87d024a5a851ce1a8fde75e9a8bdcbbb0 Oct 31 19:49:34.911309 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.911276 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:34.911397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.911353 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:34.911397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:34.911385 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:35.110939 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.110913 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f"] Oct 31 19:49:35.112333 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:49:35.112308 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5137bc_2028_4ac9_bc02_354efba8d848.slice/crio-20516df59e95a64c06f0c3ba5cb2179c17d974867f344245e72b4b96dc552312 WatchSource:0}: Error finding container 20516df59e95a64c06f0c3ba5cb2179c17d974867f344245e72b4b96dc552312: Status 404 returned error can't find the container with id 20516df59e95a64c06f0c3ba5cb2179c17d974867f344245e72b4b96dc552312 Oct 31 19:49:35.115019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.114958 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:35.115120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.115050 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:35.115120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.115081 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:35.472016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.464036 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:49:35.488832 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.488797 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" event={"ID":"fe5137bc-2028-4ac9-bc02-354efba8d848","Type":"ContainerStarted","Data":"3a20256037e440287cc58e129d4690af8928c62f620b8f7fdc06586cb53d1cbb"} Oct 31 19:49:35.489117 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.489096 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" event={"ID":"fe5137bc-2028-4ac9-bc02-354efba8d848","Type":"ContainerStarted","Data":"20516df59e95a64c06f0c3ba5cb2179c17d974867f344245e72b4b96dc552312"} Oct 31 19:49:35.492326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.492294 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" event={"ID":"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25","Type":"ContainerStarted","Data":"01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad"} Oct 31 19:49:35.492542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.492521 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" event={"ID":"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25","Type":"ContainerStarted","Data":"c7eee9d8bffcbf707d0d9477834bfff87d024a5a851ce1a8fde75e9a8bdcbbb0"} Oct 31 19:49:35.721624 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.721569 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podStartSLOduration=1.72155303 podStartE2EDuration="1.72155303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:35.69744747 +0000 UTC m=+1946.937670600" watchObservedRunningTime="2026-10-31 19:49:35.72155303 +0000 UTC m=+1946.961776158" Oct 31 19:49:35.742439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.742354 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:35.818197 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:35.818128 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podStartSLOduration=1.818105839 podStartE2EDuration="1.818105839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:35.805271561 +0000 UTC m=+1947.045494691" watchObservedRunningTime="2026-10-31 19:49:35.818105839 +0000 UTC m=+1947.058328969" Oct 31 19:49:36.498472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:36.498440 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:36.503361 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:36.503330 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:49:36.503595 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:36.503569 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 19:49:36.744543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:36.744499 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:49:36.744753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:36.744555 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 19:49:37.502352 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:37.501893 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:49:37.502352 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:37.501949 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 19:49:37.742568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:37.742524 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:49:37.742771 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:37.742583 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 19:49:38.494493 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:38.494453 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:49:38.494792 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:38.494759 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 19:49:38.741986 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:38.741951 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:49:38.742460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:38.742080 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 19:49:39.488587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:39.488549 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:49:39.488924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:39.488877 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 19:49:39.742968 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:39.742886 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:49:39.742968 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:39.742944 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 19:49:40.488575 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:40.488524 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:49:40.488885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:40.488837 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 19:49:40.742430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:40.742350 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:49:40.742430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:40.742400 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 19:49:41.494326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:41.493126 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:49:41.494326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:41.493213 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 19:49:41.742728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:41.742693 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:49:41.742918 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:41.742755 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 19:49:42.489044 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:42.489010 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:49:42.489300 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:42.489261 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 19:49:42.742312 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:42.742228 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:49:42.742799 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:42.742320 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 19:49:43.843071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:43.843032 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:43.844704 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:43.844674 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:43.845889 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:43.845844 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 19:49:44.512953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:44.512906 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:44.513319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:44.513289 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:44.514268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:44.514238 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" Oct 31 19:49:47.450710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.450676 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw"] Oct 31 19:49:47.457439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.457417 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.552119 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.552080 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw"] Oct 31 19:49:47.589341 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.589306 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-31a1ba53-serving-cert\"" Oct 31 19:49:47.656543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.656502 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f247a99-5beb-4c36-bd0e-f742a4539d17-openshift-service-ca\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.656915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.656889 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-maas-api-tls\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.657161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.657140 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tcz\" (UniqueName: \"kubernetes.io/projected/4f247a99-5beb-4c36-bd0e-f742a4539d17-kube-api-access-25tcz\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.657335 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.657318 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-metrics-certs\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.808057 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.795323 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-maas-api-tls\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.816016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.808435 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tcz\" (UniqueName: \"kubernetes.io/projected/4f247a99-5beb-4c36-bd0e-f742a4539d17-kube-api-access-25tcz\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.878019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.852804 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-metrics-certs\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.911621 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.878680 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f247a99-5beb-4c36-bd0e-f742a4539d17-openshift-service-ca\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:47.912127 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:47.885609 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f247a99-5beb-4c36-bd0e-f742a4539d17-openshift-service-ca\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:48.071866 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:48.046783 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-metrics-certs\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:48.081730 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:48.073756 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-maas-api-tls\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:48.133910 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:48.133817 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tcz\" (UniqueName: \"kubernetes.io/projected/4f247a99-5beb-4c36-bd0e-f742a4539d17-kube-api-access-25tcz\") pod \"maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:48.146763 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:48.134279 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:48.754317 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:48.754287 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw"] Oct 31 19:49:48.755936 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:49:48.755902 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f247a99_5beb_4c36_bd0e_f742a4539d17.slice/crio-8cc341964bd80986f7ca231e89b49ffb9b0e8b68362ec8121d29dbde7a17e9d7 WatchSource:0}: Error finding container 8cc341964bd80986f7ca231e89b49ffb9b0e8b68362ec8121d29dbde7a17e9d7: Status 404 returned error can't find the container with id 8cc341964bd80986f7ca231e89b49ffb9b0e8b68362ec8121d29dbde7a17e9d7 Oct 31 19:49:48.901060 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:48.901020 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" event={"ID":"4f247a99-5beb-4c36-bd0e-f742a4539d17","Type":"ContainerStarted","Data":"8cc341964bd80986f7ca231e89b49ffb9b0e8b68362ec8121d29dbde7a17e9d7"} Oct 31 19:49:49.269020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.267429 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv"] Oct 31 19:49:49.294019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.277239 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.399288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.399239 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.399510 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.399308 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.399510 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.399454 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb1bd7d-63fa-43ed-8674-9299055d81c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.399770 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.399523 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.399770 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.399583 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.400149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.400121 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.400240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.400185 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.400240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.400229 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkjn\" (UniqueName: \"kubernetes.io/projected/afb1bd7d-63fa-43ed-8674-9299055d81c9-kube-api-access-vwkjn\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.400352 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.400278 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.498887 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.498853 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv"] Oct 31 19:49:49.502519 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502491 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.502656 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502544 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.502656 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502587 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkjn\" (UniqueName: \"kubernetes.io/projected/afb1bd7d-63fa-43ed-8674-9299055d81c9-kube-api-access-vwkjn\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.502656 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502637 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.502828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502753 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.502828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502794 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.502932 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502904 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb1bd7d-63fa-43ed-8674-9299055d81c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.503014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502940 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.503014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502982 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.503135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.503103 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.503258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.503232 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.503332 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.502943 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.503523 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.503504 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.503695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.503673 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb1bd7d-63fa-43ed-8674-9299055d81c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.505490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.505467 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.505655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.505638 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.555532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.555458 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb1bd7d-63fa-43ed-8674-9299055d81c9-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.555653 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.555604 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkjn\" (UniqueName: \"kubernetes.io/projected/afb1bd7d-63fa-43ed-8674-9299055d81c9-kube-api-access-vwkjn\") pod \"maas-default-gateway-openshift-default-58d898bd55-mdfhv\" (UID: \"afb1bd7d-63fa-43ed-8674-9299055d81c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:49.606184 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:49.606153 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:50.044174 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:50.044131 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv"] Oct 31 19:49:50.045482 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:49:50.045448 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb1bd7d_63fa_43ed_8674_9299055d81c9.slice/crio-812debd8c8bff5822659f844c2e8dadb9d2de9b740a6a7fec2b4f67c160a48af WatchSource:0}: Error finding container 812debd8c8bff5822659f844c2e8dadb9d2de9b740a6a7fec2b4f67c160a48af: Status 404 returned error can't find the container with id 812debd8c8bff5822659f844c2e8dadb9d2de9b740a6a7fec2b4f67c160a48af Oct 31 19:49:50.049556 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:50.049514 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:50.049641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:50.049615 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:50.049695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:50.049664 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:51.043276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.041742 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" event={"ID":"4f247a99-5beb-4c36-bd0e-f742a4539d17","Type":"ContainerStarted","Data":"8f6cad01a65d52e2ee8781dc3b775c49bd4f4459456b0f4ce5f7f551979bec5b"} Oct 31 19:49:51.043276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.041833 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:51.044139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.044110 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" event={"ID":"afb1bd7d-63fa-43ed-8674-9299055d81c9","Type":"ContainerStarted","Data":"6aadab6294af4a7c09cd9782230c96eddb944d8a0354f614eb58f5ba37d68a65"} Oct 31 19:49:51.044223 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.044162 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" event={"ID":"afb1bd7d-63fa-43ed-8674-9299055d81c9","Type":"ContainerStarted","Data":"812debd8c8bff5822659f844c2e8dadb9d2de9b740a6a7fec2b4f67c160a48af"} Oct 31 19:49:51.159407 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.159354 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" podStartSLOduration=2.9979581509999997 podStartE2EDuration="4.15933815s" totalImagesPullingTime="1.161379999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:51.157899881 +0000 UTC m=+1962.398123004" watchObservedRunningTime="2026-10-31 19:49:51.15933815 +0000 UTC m=+1962.399561280" Oct 31 19:49:51.321136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.321025 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podStartSLOduration=2.320985849 podStartE2EDuration="2.320985849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:51.319836029 +0000 UTC m=+1962.560059159" watchObservedRunningTime="2026-10-31 19:49:51.320985849 +0000 UTC m=+1962.561208978" Oct 31 19:49:51.611610 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.611523 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:51.611950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.611923 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:51.622454 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:51.622420 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:52.622037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:52.621312 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:52.622037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:52.621386 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:53.607205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:53.607173 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:53.607351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:53.607228 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:54.606679 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:54.606638 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:54.607241 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:54.606703 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:55.606803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:55.606767 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:55.607205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:55.606826 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:56.606777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:56.606727 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:56.606956 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:56.606802 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:57.083109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:57.083080 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:49:57.329146 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:49:57.329104 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:49:57.606592 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:57.606562 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:57.606752 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:57.606633 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:58.148705 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.148650 6481 generic.go:356] "Generic (PLEG): container finished" podID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerID="b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82" exitCode=0 Oct 31 19:49:58.149821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.149643 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" event={"ID":"2f56e5ba-5936-4ca9-b862-2064e8bb1f09","Type":"ContainerDied","Data":"b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82"} Oct 31 19:49:58.160034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.156466 6481 scope.go:122] "RemoveContainer" containerID="b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82" Oct 31 19:49:58.166749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.162423 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:58.166749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.162527 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:58.166749 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.162580 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:49:58.607441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.607404 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:49:58.607690 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.607470 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 19:49:58.713394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:58.713357 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:49:59.205358 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:59.205324 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" event={"ID":"2f56e5ba-5936-4ca9-b862-2064e8bb1f09","Type":"ContainerStarted","Data":"bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda"} Oct 31 19:49:59.731345 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:59.731216 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:59.732088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:59.732063 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:59.733790 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:59.733764 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" Oct 31 19:49:59.930747 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:59.930713 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:49:59.931903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:59.931869 6481 patch_prober.go:29] interesting pod/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 19:49:59.932075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:49:59.931942 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 19:50:00.930398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:00.930365 6481 patch_prober.go:29] interesting pod/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 19:50:00.930799 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:00.930424 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 19:50:01.930553 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:01.930513 6481 patch_prober.go:29] interesting pod/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 19:50:01.931096 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:01.930580 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 19:50:02.930904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:02.930869 6481 patch_prober.go:29] interesting pod/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 19:50:02.931324 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:02.930922 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 19:50:03.931186 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:03.931151 6481 patch_prober.go:29] interesting pod/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 19:50:03.931584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:03.931207 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 19:50:04.935877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:04.935840 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:50:04.936353 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:04.936242 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:50:04.936961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:04.936941 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:50:27.502299 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:50:27.502248 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:50:29.475097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:29.475061 6481 generic.go:356] "Generic (PLEG): container finished" podID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerID="b1f99b67b77b61dacb5fb619020081240687b469c9e45982a736241e0f5f68f7" exitCode=0 Oct 31 19:50:29.475525 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:29.475133 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" event={"ID":"9a7724f7-e34e-4212-9805-6c09fc227c18","Type":"ContainerDied","Data":"b1f99b67b77b61dacb5fb619020081240687b469c9e45982a736241e0f5f68f7"} Oct 31 19:50:29.475627 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:29.475612 6481 scope.go:122] "RemoveContainer" containerID="b1f99b67b77b61dacb5fb619020081240687b469c9e45982a736241e0f5f68f7" Oct 31 19:50:29.477235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:29.477205 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:50:29.477354 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:29.477288 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:50:29.477354 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:29.477336 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:50:30.498307 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:30.498273 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" event={"ID":"9a7724f7-e34e-4212-9805-6c09fc227c18","Type":"ContainerStarted","Data":"eb50857c6b82d926ebd266c6d5da038fb746ea59563ec2b101ce5dba5d3d9bd1"} Oct 31 19:50:30.525182 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:30.525147 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:50:30.526288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:30.526260 6481 patch_prober.go:29] interesting pod/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 19:50:30.526475 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:30.526328 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 19:50:31.525355 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:31.525321 6481 patch_prober.go:29] interesting pod/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 19:50:31.525761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:31.525386 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 19:50:32.524921 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:32.524888 6481 patch_prober.go:29] interesting pod/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 19:50:32.525129 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:32.524950 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 19:50:33.524583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:33.524551 6481 patch_prober.go:29] interesting pod/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 19:50:33.525037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:33.524608 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 19:50:34.003594 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:34.003559 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:50:34.524427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:34.524394 6481 patch_prober.go:29] interesting pod/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 19:50:34.524636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:34.524451 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 19:50:35.524906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:35.524871 6481 patch_prober.go:29] interesting pod/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 19:50:35.525379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:35.524928 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 19:50:36.528794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:36.528760 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:50:36.529708 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:50:36.529685 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:50:57.466020 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:50:57.465210 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:51:19.507394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:19.507357 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv"] Oct 31 19:51:19.507901 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:19.507545 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" podUID="df04dc87-6cba-4037-9724-c5d66903d7dd" containerName="maas-api" containerID="cri-o://cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf" gracePeriod=30 Oct 31 19:51:19.938405 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:19.938171 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:51:20.078024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.053614 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls\") pod \"df04dc87-6cba-4037-9724-c5d66903d7dd\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " Oct 31 19:51:20.078024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.053815 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-metrics-certs\") pod \"df04dc87-6cba-4037-9724-c5d66903d7dd\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " Oct 31 19:51:20.078024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.054032 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df04dc87-6cba-4037-9724-c5d66903d7dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df04dc87-6cba-4037-9724-c5d66903d7dd-openshift-service-ca\") pod \"df04dc87-6cba-4037-9724-c5d66903d7dd\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " Oct 31 19:51:20.078024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.054118 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df04dc87-6cba-4037-9724-c5d66903d7dd-kube-api-access-rbfr8\" (UniqueName: \"kubernetes.io/projected/df04dc87-6cba-4037-9724-c5d66903d7dd-kube-api-access-rbfr8\") pod \"df04dc87-6cba-4037-9724-c5d66903d7dd\" (UID: \"df04dc87-6cba-4037-9724-c5d66903d7dd\") " Oct 31 19:51:20.079282 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.079228 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df04dc87-6cba-4037-9724-c5d66903d7dd-openshift-service-ca" pod "df04dc87-6cba-4037-9724-c5d66903d7dd" (UID: "df04dc87-6cba-4037-9724-c5d66903d7dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:20.104965 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.102636 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-metrics-certs" pod "df04dc87-6cba-4037-9724-c5d66903d7dd" (UID: "df04dc87-6cba-4037-9724-c5d66903d7dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:20.143787 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.143694 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df04dc87-6cba-4037-9724-c5d66903d7dd-kube-api-access-rbfr8" pod "df04dc87-6cba-4037-9724-c5d66903d7dd" (UID: "df04dc87-6cba-4037-9724-c5d66903d7dd"). InnerVolumeSpecName "kube-api-access-rbfr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:20.151546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.151510 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls" pod "df04dc87-6cba-4037-9724-c5d66903d7dd" (UID: "df04dc87-6cba-4037-9724-c5d66903d7dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:20.158835 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.158740 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:20.158835 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.158778 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df04dc87-6cba-4037-9724-c5d66903d7dd-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:20.158835 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.158794 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbfr8\" (UniqueName: \"kubernetes.io/projected/df04dc87-6cba-4037-9724-c5d66903d7dd-kube-api-access-rbfr8\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:20.158835 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.158809 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df04dc87-6cba-4037-9724-c5d66903d7dd-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:20.280015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.277016 6481 generic.go:356] "Generic (PLEG): container finished" podID="df04dc87-6cba-4037-9724-c5d66903d7dd" containerID="cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf" exitCode=0 Oct 31 19:51:20.280015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.277191 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" event={"ID":"df04dc87-6cba-4037-9724-c5d66903d7dd","Type":"ContainerDied","Data":"cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf"} Oct 31 19:51:20.280015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.277243 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" event={"ID":"df04dc87-6cba-4037-9724-c5d66903d7dd","Type":"ContainerDied","Data":"9c025bf71eae1d012a0fc4d084f1142c5ca065f7a67b54cf2c7077cd21119554"} Oct 31 19:51:20.280015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.277274 6481 scope.go:122] "RemoveContainer" containerID="cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf" Oct 31 19:51:20.280015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.277608 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv" Oct 31 19:51:20.303171 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.303149 6481 scope.go:122] "RemoveContainer" containerID="cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf" Oct 31 19:51:20.303468 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:51:20.303442 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf\": container with ID starting with cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf not found: ID does not exist" containerID="cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf" Oct 31 19:51:20.303563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.303473 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf"} err="failed to get container status \"cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf\": rpc error: code = NotFound desc = could not find container \"cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf\": container with ID starting with cfd7a17cbbf10cdc7f15640bc992396d999bae348cc4005bef6c9fb68e248baf not found: ID does not exist" Oct 31 19:51:20.437960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.437859 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv"] Oct 31 19:51:20.483221 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.483185 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9be7bd13-864c9cf8d4-nzvvv"] Oct 31 19:51:20.881595 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:20.881552 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df04dc87-6cba-4037-9724-c5d66903d7dd" path="/var/lib/kubelet/pods/df04dc87-6cba-4037-9724-c5d66903d7dd/volumes" Oct 31 19:51:21.962578 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:21.962542 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq"] Oct 31 19:51:21.963158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:21.962837 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" containerID="cri-o://bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda" gracePeriod=30 Oct 31 19:51:27.240739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.240682 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:51:27.304111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304075 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-envoy\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.304277 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304199 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-podinfo\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.304277 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304243 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-credential-socket\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.304277 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304272 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istiod-ca-cert\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.304459 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304386 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-kube-api-access-tkfx4\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-kube-api-access-tkfx4\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.304512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304498 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-socket\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.305065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304559 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-token\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-token\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.305065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304665 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-data\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.305065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304731 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-certs\") pod \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\" (UID: \"2f56e5ba-5936-4ca9-b862-2064e8bb1f09\") " Oct 31 19:51:27.305065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304691 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-credential-socket" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:27.305065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304774 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istiod-ca-cert" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:27.305065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.304941 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-data" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:27.305065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.305028 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-socket" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:27.305543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.305081 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-certs" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:27.305743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.305550 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.305743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.305573 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.305743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.305589 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.305743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.305602 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.305743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.305617 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.306666 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.306639 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-envoy" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:27.307191 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.307160 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-kube-api-access-tkfx4" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "kube-api-access-tkfx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:27.307398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.307371 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-podinfo" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:27.308295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.308274 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-token" pod "2f56e5ba-5936-4ca9-b862-2064e8bb1f09" (UID: "2f56e5ba-5936-4ca9-b862-2064e8bb1f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:27.394026 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:51:27.393590 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:51:27.453404 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.429793 6481 generic.go:356] "Generic (PLEG): container finished" podID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerID="bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda" exitCode=0 Oct 31 19:51:27.453404 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.430085 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" event={"ID":"2f56e5ba-5936-4ca9-b862-2064e8bb1f09","Type":"ContainerDied","Data":"bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda"} Oct 31 19:51:27.453404 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.430137 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" event={"ID":"2f56e5ba-5936-4ca9-b862-2064e8bb1f09","Type":"ContainerDied","Data":"808ed7f3f7c8ecd94e2c9225c9084df86f047c55c44558e39ef52a5109a084ee"} Oct 31 19:51:27.453404 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.430166 6481 scope.go:122] "RemoveContainer" containerID="bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda" Oct 31 19:51:27.453404 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.430340 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq" Oct 31 19:51:27.494424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.455509 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkfx4\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-kube-api-access-tkfx4\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.494424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.455544 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.494424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.455562 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.494424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.455579 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f56e5ba-5936-4ca9-b862-2064e8bb1f09-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.628494 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.628106 6481 scope.go:122] "RemoveContainer" containerID="b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82" Oct 31 19:51:27.672663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.672618 6481 scope.go:122] "RemoveContainer" containerID="bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda" Oct 31 19:51:27.673364 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:51:27.673322 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda\": container with ID starting with bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda not found: ID does not exist" containerID="bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda" Oct 31 19:51:27.673771 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.673597 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda"} err="failed to get container status \"bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda\": rpc error: code = NotFound desc = could not find container \"bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda\": container with ID starting with bb48720e5d0b621eb3530d9831c6a418c46a2b8d6e1cd0b62b3f45dca3506cda not found: ID does not exist" Oct 31 19:51:27.674045 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.673976 6481 scope.go:122] "RemoveContainer" containerID="b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82" Oct 31 19:51:27.674980 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:51:27.674949 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82\": container with ID starting with b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82 not found: ID does not exist" containerID="b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82" Oct 31 19:51:27.675096 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.675012 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82"} err="failed to get container status \"b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82\": rpc error: code = NotFound desc = could not find container \"b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82\": container with ID starting with b20b9e23c4e8d338458080ca81ad5b7b9add356b1fd42cf5ec6d9059e5b3cb82 not found: ID does not exist" Oct 31 19:51:27.714446 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.714414 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq"] Oct 31 19:51:27.719006 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:27.718965 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9be7bd13-openshift-default-79bd6746b7-hlzbq"] Oct 31 19:51:28.881498 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:28.881469 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" path="/var/lib/kubelet/pods/2f56e5ba-5936-4ca9-b862-2064e8bb1f09/volumes" Oct 31 19:51:31.511276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:31.511209 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8"] Oct 31 19:51:31.511779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:31.511452 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" containerID="cri-o://eb50857c6b82d926ebd266c6d5da038fb746ea59563ec2b101ce5dba5d3d9bd1" gracePeriod=30 Oct 31 19:51:32.630479 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.630449 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl"] Oct 31 19:51:32.631158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631142 6481 cpu_manager.go:414] "containerMap: removing container" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" Oct 31 19:51:32.631219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631160 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" Oct 31 19:51:32.631219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631173 6481 cpu_manager.go:414] "containerMap: removing container" podUID="df04dc87-6cba-4037-9724-c5d66903d7dd" containerName="maas-api" Oct 31 19:51:32.631219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631178 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df04dc87-6cba-4037-9724-c5d66903d7dd" containerName="maas-api" Oct 31 19:51:32.631219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631208 6481 cpu_manager.go:414] "containerMap: removing container" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" Oct 31 19:51:32.631219 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631214 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" Oct 31 19:51:32.631365 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631307 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" Oct 31 19:51:32.631365 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.631321 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="df04dc87-6cba-4037-9724-c5d66903d7dd" containerName="maas-api" Oct 31 19:51:32.634734 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.634708 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.693079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.693046 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl"] Oct 31 19:51:32.701931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.701900 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-data\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702137 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.701967 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istiod-ca-cert\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.702142 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j89xz\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-kube-api-access-j89xz\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.702186 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-credential-socket\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.702284 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-token\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.702330 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-podinfo\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702444 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.702428 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-socket\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.702515 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-envoy\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.702639 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.702624 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-certs\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.731938 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.731906 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-84fd8e82-openshift-default-dockercfg-vbk2j\"" Oct 31 19:51:32.809579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.809538 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-envoy\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.815814 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.815778 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-certs\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.816124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.816100 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-data\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.816281 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.816264 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istiod-ca-cert\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.816460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.816413 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89xz\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-kube-api-access-j89xz\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.816581 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.816567 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-credential-socket\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.816761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.816748 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-token\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.816884 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.816872 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-podinfo\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.817089 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.817074 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-socket\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.817678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.817658 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-socket\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.820275 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.818350 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-credential-socket\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.820414 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.819033 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-certs\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.820507 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.819705 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istiod-ca-cert\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.820595 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.819956 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-data\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.857018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.856673 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-podinfo\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.933436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.933306 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-envoy\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.947969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.947935 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89xz\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-kube-api-access-j89xz\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:32.954177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:32.954129 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-token\") pod \"e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:33.252448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.252412 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:33.351543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.350530 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw"] Oct 31 19:51:33.351543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.350757 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" podUID="4f247a99-5beb-4c36-bd0e-f742a4539d17" containerName="maas-api" containerID="cri-o://8f6cad01a65d52e2ee8781dc3b775c49bd4f4459456b0f4ce5f7f551979bec5b" gracePeriod=30 Oct 31 19:51:33.500912 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.500881 6481 generic.go:356] "Generic (PLEG): container finished" podID="4f247a99-5beb-4c36-bd0e-f742a4539d17" containerID="8f6cad01a65d52e2ee8781dc3b775c49bd4f4459456b0f4ce5f7f551979bec5b" exitCode=0 Oct 31 19:51:33.501119 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.500947 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" event={"ID":"4f247a99-5beb-4c36-bd0e-f742a4539d17","Type":"ContainerDied","Data":"8f6cad01a65d52e2ee8781dc3b775c49bd4f4459456b0f4ce5f7f551979bec5b"} Oct 31 19:51:33.747252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.747224 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:51:33.795065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.789106 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl"] Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.815736 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f247a99-5beb-4c36-bd0e-f742a4539d17-kube-api-access-25tcz\" (UniqueName: \"kubernetes.io/projected/4f247a99-5beb-4c36-bd0e-f742a4539d17-kube-api-access-25tcz\") pod \"4f247a99-5beb-4c36-bd0e-f742a4539d17\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.815911 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-maas-api-tls\") pod \"4f247a99-5beb-4c36-bd0e-f742a4539d17\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.816048 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-metrics-certs\") pod \"4f247a99-5beb-4c36-bd0e-f742a4539d17\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.816154 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f247a99-5beb-4c36-bd0e-f742a4539d17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f247a99-5beb-4c36-bd0e-f742a4539d17-openshift-service-ca\") pod \"4f247a99-5beb-4c36-bd0e-f742a4539d17\" (UID: \"4f247a99-5beb-4c36-bd0e-f742a4539d17\") " Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.820512 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.820636 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.820680 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:51:33.829710 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.829619 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f247a99-5beb-4c36-bd0e-f742a4539d17-openshift-service-ca" pod "4f247a99-5beb-4c36-bd0e-f742a4539d17" (UID: "4f247a99-5beb-4c36-bd0e-f742a4539d17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:33.870338 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.864327 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f247a99-5beb-4c36-bd0e-f742a4539d17-kube-api-access-25tcz" pod "4f247a99-5beb-4c36-bd0e-f742a4539d17" (UID: "4f247a99-5beb-4c36-bd0e-f742a4539d17"). InnerVolumeSpecName "kube-api-access-25tcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:33.932628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.932587 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f247a99-5beb-4c36-bd0e-f742a4539d17-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:33.932628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:33.932630 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tcz\" (UniqueName: \"kubernetes.io/projected/4f247a99-5beb-4c36-bd0e-f742a4539d17-kube-api-access-25tcz\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.087082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.072495 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-maas-api-tls" pod "4f247a99-5beb-4c36-bd0e-f742a4539d17" (UID: "4f247a99-5beb-4c36-bd0e-f742a4539d17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:34.087082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.072617 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-metrics-certs" pod "4f247a99-5beb-4c36-bd0e-f742a4539d17" (UID: "4f247a99-5beb-4c36-bd0e-f742a4539d17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:34.166412 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.166382 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.166412 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.166409 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f247a99-5beb-4c36-bd0e-f742a4539d17-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.559347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.559295 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" event={"ID":"4f247a99-5beb-4c36-bd0e-f742a4539d17","Type":"ContainerDied","Data":"8cc341964bd80986f7ca231e89b49ffb9b0e8b68362ec8121d29dbde7a17e9d7"} Oct 31 19:51:34.559684 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.559620 6481 scope.go:122] "RemoveContainer" containerID="8f6cad01a65d52e2ee8781dc3b775c49bd4f4459456b0f4ce5f7f551979bec5b" Oct 31 19:51:34.559986 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.559969 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw" Oct 31 19:51:34.643127 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.639836 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" event={"ID":"c1d23075-4d02-4dba-83fb-6a7e56a0f31c","Type":"ContainerStarted","Data":"d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3"} Oct 31 19:51:34.643127 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.639900 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" event={"ID":"c1d23075-4d02-4dba-83fb-6a7e56a0f31c","Type":"ContainerStarted","Data":"14c2bf5b43c688296b24d0e16bc8130c6eb2f9115751ce66cf667e8fad4f5c50"} Oct 31 19:51:34.793971 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.793904 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podStartSLOduration=2.793881564 podStartE2EDuration="2.793881564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:34.790247601 +0000 UTC m=+2066.030470745" watchObservedRunningTime="2026-10-31 19:51:34.793881564 +0000 UTC m=+2066.034104753" Oct 31 19:51:34.917325 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.917238 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw"] Oct 31 19:51:34.971121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:34.971072 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-31a1ba53-7bcf7c6755-t72qw"] Oct 31 19:51:35.254008 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:35.253962 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:35.259906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:35.259881 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:35.669068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:35.663385 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:35.670096 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:35.669701 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:51:36.544265 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.542715 6481 patch_prober.go:29] interesting pod/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 19:51:36.544265 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.542813 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 19:51:36.674893 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.674859 6481 generic.go:356] "Generic (PLEG): container finished" podID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerID="eb50857c6b82d926ebd266c6d5da038fb746ea59563ec2b101ce5dba5d3d9bd1" exitCode=0 Oct 31 19:51:36.675081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.674931 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" event={"ID":"9a7724f7-e34e-4212-9805-6c09fc227c18","Type":"ContainerDied","Data":"eb50857c6b82d926ebd266c6d5da038fb746ea59563ec2b101ce5dba5d3d9bd1"} Oct 31 19:51:36.675081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.675005 6481 scope.go:122] "RemoveContainer" containerID="b1f99b67b77b61dacb5fb619020081240687b469c9e45982a736241e0f5f68f7" Oct 31 19:51:36.805462 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.805386 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:51:36.881400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.881367 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f247a99-5beb-4c36-bd0e-f742a4539d17" path="/var/lib/kubelet/pods/4f247a99-5beb-4c36-bd0e-f742a4539d17/volumes" Oct 31 19:51:36.918473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918428 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-certs\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.918709 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918506 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-socket\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.918709 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918564 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-data\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.918709 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918667 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-envoy\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.918936 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918710 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-podinfo\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.918936 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918749 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-kube-api-access-sp84g\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-kube-api-access-sp84g\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.918936 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918836 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7724f7-e34e-4212-9805-6c09fc227c18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7724f7-e34e-4212-9805-6c09fc227c18-istiod-ca-cert\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.919100 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.918947 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-token\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.919100 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.919079 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-credential-socket\") pod \"9a7724f7-e34e-4212-9805-6c09fc227c18\" (UID: \"9a7724f7-e34e-4212-9805-6c09fc227c18\") " Oct 31 19:51:36.920473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.919334 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-certs" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:36.920473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.919616 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-credential-socket" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:36.920473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.920054 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:36.920473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.920076 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:36.920777 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.920676 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7724f7-e34e-4212-9805-6c09fc227c18-istiod-ca-cert" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:36.920970 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.920943 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-data" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:36.923102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.922250 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-socket" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:36.923642 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.923546 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-kube-api-access-sp84g" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "kube-api-access-sp84g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:36.923642 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.923607 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-envoy" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:36.924435 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.924409 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-token" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:36.925724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:36.925700 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-podinfo" pod "9a7724f7-e34e-4212-9805-6c09fc227c18" (UID: "9a7724f7-e34e-4212-9805-6c09fc227c18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:37.021980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.021950 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:37.021980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.021980 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:37.022205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.022034 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp84g\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-kube-api-access-sp84g\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:37.022205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.022047 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7724f7-e34e-4212-9805-6c09fc227c18-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:37.022205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.022059 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:37.022205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.022073 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:37.022205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.022084 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7724f7-e34e-4212-9805-6c09fc227c18-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:37.684698 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.684669 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" Oct 31 19:51:37.685211 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.684716 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8" event={"ID":"9a7724f7-e34e-4212-9805-6c09fc227c18","Type":"ContainerDied","Data":"c62e2817f7fd2abc5cfad6533321a0385de6ba15269a3316deb59fd3d1558aa7"} Oct 31 19:51:37.685211 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.684770 6481 scope.go:122] "RemoveContainer" containerID="eb50857c6b82d926ebd266c6d5da038fb746ea59563ec2b101ce5dba5d3d9bd1" Oct 31 19:51:37.884652 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.884616 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8"] Oct 31 19:51:37.930790 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:37.930754 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ae778663-openshift-default-76cbdb85f-5mht8"] Oct 31 19:51:38.889618 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:51:38.889576 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" path="/var/lib/kubelet/pods/9a7724f7-e34e-4212-9805-6c09fc227c18/volumes" Oct 31 19:51:57.329494 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:51:57.329407 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:52:11.265515 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.265482 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9"] Oct 31 19:52:11.266146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266130 6481 cpu_manager.go:414] "containerMap: removing container" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" Oct 31 19:52:11.266146 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266146 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" Oct 31 19:52:11.266268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266166 6481 cpu_manager.go:414] "containerMap: removing container" podUID="4f247a99-5beb-4c36-bd0e-f742a4539d17" containerName="maas-api" Oct 31 19:52:11.266268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266172 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f247a99-5beb-4c36-bd0e-f742a4539d17" containerName="maas-api" Oct 31 19:52:11.266268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266182 6481 cpu_manager.go:414] "containerMap: removing container" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" Oct 31 19:52:11.266268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266187 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" Oct 31 19:52:11.266408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266347 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f56e5ba-5936-4ca9-b862-2064e8bb1f09" containerName="istio-proxy" Oct 31 19:52:11.266408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266371 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" Oct 31 19:52:11.266408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266385 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f247a99-5beb-4c36-bd0e-f742a4539d17" containerName="maas-api" Oct 31 19:52:11.266408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.266395 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7724f7-e34e-4212-9805-6c09fc227c18" containerName="istio-proxy" Oct 31 19:52:11.271597 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.271578 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.321722 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.321698 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-abb7688a-serving-cert\"" Oct 31 19:52:11.374173 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.374134 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9"] Oct 31 19:52:11.409984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.409944 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8tv\" (UniqueName: \"kubernetes.io/projected/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-kube-api-access-vd8tv\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.410201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.410109 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-metrics-certs\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.410271 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.410200 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-maas-api-tls\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.410347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.410327 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-openshift-service-ca\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.542080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.531228 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8tv\" (UniqueName: \"kubernetes.io/projected/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-kube-api-access-vd8tv\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.565915 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.565882 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-metrics-certs\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.577276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.577246 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-maas-api-tls\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.578164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.578078 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-openshift-service-ca\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.579555 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.579192 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-openshift-service-ca\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.582409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.582387 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-metrics-certs\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.583061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.583038 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-maas-api-tls\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.600074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.600040 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8tv\" (UniqueName: \"kubernetes.io/projected/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-kube-api-access-vd8tv\") pod \"maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:11.886417 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:11.886338 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:12.256882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:12.256854 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9"] Oct 31 19:52:12.258544 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:52:12.258507 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f5b6c1_f081_4a89_94a6_a6b82ab64cf3.slice/crio-705c9f798b245b7d11de3a1d2d17d561093847d1dac3811e622febb7cfc5e64c WatchSource:0}: Error finding container 705c9f798b245b7d11de3a1d2d17d561093847d1dac3811e622febb7cfc5e64c: Status 404 returned error can't find the container with id 705c9f798b245b7d11de3a1d2d17d561093847d1dac3811e622febb7cfc5e64c Oct 31 19:52:12.259961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:12.259944 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:52:13.152373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:13.152343 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" event={"ID":"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3","Type":"ContainerStarted","Data":"705c9f798b245b7d11de3a1d2d17d561093847d1dac3811e622febb7cfc5e64c"} Oct 31 19:52:14.160340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:14.160302 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" event={"ID":"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3","Type":"ContainerStarted","Data":"21050ce929ea535ce699f0437774fe18b818bd5774e6844ae76517b2c5185b40"} Oct 31 19:52:14.160728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:14.160369 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:14.274888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:14.274837 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" podStartSLOduration=2.429888155 podStartE2EDuration="3.274823075s" totalImagesPullingTime="844.93492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:14.270929113 +0000 UTC m=+2105.511152241" watchObservedRunningTime="2026-10-31 19:52:14.274823075 +0000 UTC m=+2105.515046207" Oct 31 19:52:20.173354 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.173325 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:52:20.550541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.550508 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l"] Oct 31 19:52:20.559019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.556713 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.633942 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.633902 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f37f6f32-openshift-default-dockercfg-m5h9b\"" Oct 31 19:52:20.642389 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.642256 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l"] Oct 31 19:52:20.672536 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.672507 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istiod-ca-cert\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.672706 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.672678 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-credential-socket\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.672906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.672884 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-envoy\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.673040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.673022 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-certs\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.673107 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.673095 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbgx\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-kube-api-access-rrbgx\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.673164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.673121 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-data\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.673164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.673153 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-podinfo\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.673284 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.673261 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-socket\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.673340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.673315 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-token\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.781022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.780258 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-envoy\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.781022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.780394 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-certs\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.781022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.780485 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbgx\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-kube-api-access-rrbgx\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.781022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.780520 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-data\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.781022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.780562 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-podinfo\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.821066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.813347 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-certs\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.821066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.813681 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-data\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.853479 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.853297 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-socket\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.870018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.780611 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-socket\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.870018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.863355 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-token\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.870018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.863444 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istiod-ca-cert\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.870018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.863480 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-credential-socket\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.870018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.863916 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-credential-socket\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.870018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.864863 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istiod-ca-cert\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.942075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.940953 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbgx\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-kube-api-access-rrbgx\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.944199 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.943457 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-envoy\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.949027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.945916 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-podinfo\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:20.994071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:20.994037 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-token\") pod \"e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:21.207074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.205220 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:21.401962 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.401930 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd"] Oct 31 19:52:21.402206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.402162 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" containerID="cri-o://00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79" gracePeriod=30 Oct 31 19:52:21.595240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.595200 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc"] Oct 31 19:52:21.600868 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.600844 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.656138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.656105 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-84fd8e82-serving-cert\"" Oct 31 19:52:21.678674 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.678638 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc"] Oct 31 19:52:21.732811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.732777 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ntd\" (UniqueName: \"kubernetes.io/projected/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-kube-api-access-b6ntd\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.732976 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.732888 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-openshift-service-ca\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.733084 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.733059 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.733200 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.733183 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-metrics-certs\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.766836 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:52:21.766807 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9dba8e_c1d3_4fd4_bb6b_d1a7312cb397.slice/crio-8d4dd3f31f5bbba8a5b64dfaac530b2fb8151d11fe8e9e83999419f569d50803 WatchSource:0}: Error finding container 8d4dd3f31f5bbba8a5b64dfaac530b2fb8151d11fe8e9e83999419f569d50803: Status 404 returned error can't find the container with id 8d4dd3f31f5bbba8a5b64dfaac530b2fb8151d11fe8e9e83999419f569d50803 Oct 31 19:52:21.769223 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.769202 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l"] Oct 31 19:52:21.769431 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.769403 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:21.769474 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.769464 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:21.769505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.769492 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:21.874067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.850408 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ntd\" (UniqueName: \"kubernetes.io/projected/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-kube-api-access-b6ntd\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.874067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.850596 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-openshift-service-ca\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.874067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.850748 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.874067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.850866 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-metrics-certs\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.874067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.855938 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-openshift-service-ca\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.874067 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:52:21.856065 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-serving-cert: secret "maas-api-e2e-ait-84fd8e82-serving-cert" not found Oct 31 19:52:21.874067 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:52:21.856125 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls podName:7ff3d4c5-512f-4be0-b799-2d2d1527bdee nodeName:}" failed. No retries permitted until 2026-10-31 19:52:22.356105885 +0000 UTC m=+2113.596329006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls") pod "maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" (UID: "7ff3d4c5-512f-4be0-b799-2d2d1527bdee") : secret "maas-api-e2e-ait-84fd8e82-serving-cert" not found Oct 31 19:52:21.930745 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.923511 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-metrics-certs\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:21.930745 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:21.925829 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ntd\" (UniqueName: \"kubernetes.io/projected/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-kube-api-access-b6ntd\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:22.228883 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:22.228845 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" event={"ID":"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397","Type":"ContainerStarted","Data":"5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf"} Oct 31 19:52:22.229366 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:22.228900 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" event={"ID":"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397","Type":"ContainerStarted","Data":"8d4dd3f31f5bbba8a5b64dfaac530b2fb8151d11fe8e9e83999419f569d50803"} Oct 31 19:52:22.349455 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:22.349398 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podStartSLOduration=2.349384024 podStartE2EDuration="2.349384024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:22.347451818 +0000 UTC m=+2113.587674946" watchObservedRunningTime="2026-10-31 19:52:22.349384024 +0000 UTC m=+2113.589607153" Oct 31 19:52:22.367894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:22.367863 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:22.370286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:22.370260 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls\") pod \"maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:22.524641 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:22.524527 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:23.003278 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:23.003248 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc"] Oct 31 19:52:23.004391 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:52:23.004359 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff3d4c5_512f_4be0_b799_2d2d1527bdee.slice/crio-bfb456753c80ce247ac49fa6b75564dcbad9399bad8ca773e3105d550c7dd16e WatchSource:0}: Error finding container bfb456753c80ce247ac49fa6b75564dcbad9399bad8ca773e3105d550c7dd16e: Status 404 returned error can't find the container with id bfb456753c80ce247ac49fa6b75564dcbad9399bad8ca773e3105d550c7dd16e Oct 31 19:52:23.205828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:23.205796 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:23.211221 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:23.211194 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:23.236852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:23.236820 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" event={"ID":"7ff3d4c5-512f-4be0-b799-2d2d1527bdee","Type":"ContainerStarted","Data":"bfb456753c80ce247ac49fa6b75564dcbad9399bad8ca773e3105d550c7dd16e"} Oct 31 19:52:23.237251 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:23.237117 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:23.237837 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:23.237821 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:52:24.244883 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:24.244848 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" event={"ID":"7ff3d4c5-512f-4be0-b799-2d2d1527bdee","Type":"ContainerStarted","Data":"e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400"} Oct 31 19:52:24.245368 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:24.245198 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:24.354909 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:24.354855 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" podStartSLOduration=2.411726307 podStartE2EDuration="3.354841568s" totalImagesPullingTime="943.115261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:24.353646038 +0000 UTC m=+2115.593869167" watchObservedRunningTime="2026-10-31 19:52:24.354841568 +0000 UTC m=+2115.595064697" Oct 31 19:52:26.685321 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.685290 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:52:26.784501 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784468 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-envoy\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784526 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-socket\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784552 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-token\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-token\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784573 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-data\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784594 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-kube-api-access-8tcgr\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-kube-api-access-8tcgr\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784617 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-certs\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784701 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-podinfo\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784756 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-credential-socket\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.784906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784901 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istiod-ca-cert\") pod \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\" (UID: \"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe\") " Oct 31 19:52:26.785128 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.784941 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-data" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.785128 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.785055 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-certs" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.785239 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.785203 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-credential-socket" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.785494 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.785466 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istiod-ca-cert" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:52:26.785610 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.785494 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.785610 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.785515 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.785610 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.785527 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.785756 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.785700 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-socket" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.787259 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.787232 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-envoy" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:26.787398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.787321 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-token" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:26.787509 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.787491 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-kube-api-access-8tcgr" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "kube-api-access-8tcgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:26.787724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.787700 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-podinfo" pod "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" (UID: "d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:52:26.887548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.887513 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.887548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.887548 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.887548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.887562 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.887849 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.887575 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.887849 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.887588 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.887849 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:26.887601 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tcgr\" (UniqueName: \"kubernetes.io/projected/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe-kube-api-access-8tcgr\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.277602 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.277571 6481 generic.go:356] "Generic (PLEG): container finished" podID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerID="00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79" exitCode=0 Oct 31 19:52:27.277776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.277618 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" Oct 31 19:52:27.277776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.277654 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" event={"ID":"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe","Type":"ContainerDied","Data":"00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79"} Oct 31 19:52:27.277776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.277707 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd" event={"ID":"d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe","Type":"ContainerDied","Data":"509649a422897025884d4950fc6d0976ff7930730473195c591ec71be51de88c"} Oct 31 19:52:27.277776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.277729 6481 scope.go:122] "RemoveContainer" containerID="00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79" Oct 31 19:52:27.290766 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.290747 6481 scope.go:122] "RemoveContainer" containerID="00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79" Oct 31 19:52:27.291306 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:52:27.291279 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79\": container with ID starting with 00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79 not found: ID does not exist" containerID="00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79" Oct 31 19:52:27.291396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.291316 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79"} err="failed to get container status \"00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79\": rpc error: code = NotFound desc = could not find container \"00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79\": container with ID starting with 00b3a46694f88d81c1a21ac139e677d7673b1e288c7bb0148392c5cfd0fd0c79 not found: ID does not exist" Oct 31 19:52:27.343763 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:52:27.343728 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:52:27.438380 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.438350 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd"] Oct 31 19:52:27.443684 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:27.443657 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-x66fd"] Oct 31 19:52:28.882523 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:28.882488 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" path="/var/lib/kubelet/pods/d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe/volumes" Oct 31 19:52:30.268794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:30.268766 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:52:31.131798 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.131758 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp"] Oct 31 19:52:31.133202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.133169 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" Oct 31 19:52:31.133439 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.133423 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" Oct 31 19:52:31.133706 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.133693 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4832d60-3727-4f3c-a8a8-8f38bbb0bbfe" containerName="istio-proxy" Oct 31 19:52:31.232411 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.232372 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp"] Oct 31 19:52:31.234121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.233975 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:31.244450 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.244422 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps"] Oct 31 19:52:31.317032 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.297217 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f37f6f32-dockercfg-hvtfx\"" Oct 31 19:52:31.317032 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.297327 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f37f6f32\"" Oct 31 19:52:31.370101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.369726 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps"] Oct 31 19:52:31.370523 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.370376 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.389516 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.389429 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbl5f\" (UniqueName: \"kubernetes.io/projected/cefbc76e-859f-4919-a607-2a75b90f3697-kube-api-access-xbl5f\") pod \"payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.391697 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.391664 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdh22\" (UniqueName: \"kubernetes.io/projected/6936d404-a7af-48e0-ba93-96f4cff8e734-kube-api-access-vdh22\") pod \"payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:31.392004 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.391970 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbc76e-859f-4919-a607-2a75b90f3697-plugins-config-volume\") pod \"payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.392268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.392240 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936d404-a7af-48e0-ba93-96f4cff8e734-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:31.641075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.604274 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdh22\" (UniqueName: \"kubernetes.io/projected/6936d404-a7af-48e0-ba93-96f4cff8e734-kube-api-access-vdh22\") pod \"payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:31.677343 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.677198 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbc76e-859f-4919-a607-2a75b90f3697-plugins-config-volume\") pod \"payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.677768 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.677741 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936d404-a7af-48e0-ba93-96f4cff8e734-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:31.678187 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.678168 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbl5f\" (UniqueName: \"kubernetes.io/projected/cefbc76e-859f-4919-a607-2a75b90f3697-kube-api-access-xbl5f\") pod \"payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.680108 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.680072 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbc76e-859f-4919-a607-2a75b90f3697-plugins-config-volume\") pod \"payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.681325 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.681301 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936d404-a7af-48e0-ba93-96f4cff8e734-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:31.713486 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.713430 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdh22\" (UniqueName: \"kubernetes.io/projected/6936d404-a7af-48e0-ba93-96f4cff8e734-kube-api-access-vdh22\") pod \"payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:31.732625 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.732583 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbl5f\" (UniqueName: \"kubernetes.io/projected/cefbc76e-859f-4919-a607-2a75b90f3697-kube-api-access-xbl5f\") pod \"payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.743485 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.743448 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:31.919747 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:31.919669 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:32.116290 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:32.116261 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps"] Oct 31 19:52:32.117382 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:52:32.117359 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefbc76e_859f_4919_a607_2a75b90f3697.slice/crio-dce3a6b371f9b43ba6b87fe7e547c275ba2bb4809c6042909178bdd4477c7c72 WatchSource:0}: Error finding container dce3a6b371f9b43ba6b87fe7e547c275ba2bb4809c6042909178bdd4477c7c72: Status 404 returned error can't find the container with id dce3a6b371f9b43ba6b87fe7e547c275ba2bb4809c6042909178bdd4477c7c72 Oct 31 19:52:32.288011 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:32.287968 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp"] Oct 31 19:52:32.289335 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:52:32.289311 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6936d404_a7af_48e0_ba93_96f4cff8e734.slice/crio-163272de163048c402e77b25b97cd0f760c38d6de5627fb344eba4f10def253d WatchSource:0}: Error finding container 163272de163048c402e77b25b97cd0f760c38d6de5627fb344eba4f10def253d: Status 404 returned error can't find the container with id 163272de163048c402e77b25b97cd0f760c38d6de5627fb344eba4f10def253d Oct 31 19:52:32.351951 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:32.351919 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" event={"ID":"cefbc76e-859f-4919-a607-2a75b90f3697","Type":"ContainerStarted","Data":"6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06"} Oct 31 19:52:32.352327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:32.351982 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:32.352327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:32.352019 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" event={"ID":"cefbc76e-859f-4919-a607-2a75b90f3697","Type":"ContainerStarted","Data":"dce3a6b371f9b43ba6b87fe7e547c275ba2bb4809c6042909178bdd4477c7c72"} Oct 31 19:52:32.353707 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:32.353683 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" event={"ID":"6936d404-a7af-48e0-ba93-96f4cff8e734","Type":"ContainerStarted","Data":"163272de163048c402e77b25b97cd0f760c38d6de5627fb344eba4f10def253d"} Oct 31 19:52:32.457913 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:32.457805 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" podStartSLOduration=1.457792119 podStartE2EDuration="1.457792119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:32.456362926 +0000 UTC m=+2123.696586046" watchObservedRunningTime="2026-10-31 19:52:32.457792119 +0000 UTC m=+2123.698015248" Oct 31 19:52:33.362113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:33.362041 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" event={"ID":"6936d404-a7af-48e0-ba93-96f4cff8e734","Type":"ContainerStarted","Data":"e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f"} Oct 31 19:52:33.362554 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:33.362146 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:33.473965 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:33.473911 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" podStartSLOduration=2.473896368 podStartE2EDuration="2.473896368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:33.471634147 +0000 UTC m=+2124.711857277" watchObservedRunningTime="2026-10-31 19:52:33.473896368 +0000 UTC m=+2124.714119497" Oct 31 19:52:36.488891 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:36.488849 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk"] Oct 31 19:52:36.489381 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:36.489194 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" containerID="cri-o://62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553" gracePeriod=30 Oct 31 19:52:42.141015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.138841 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:52:42.331381 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.331346 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-kube-api-access-swnnt\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-kube-api-access-swnnt\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.331706 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.331685 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-data\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.331877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.331860 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-certs\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.332038 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.332021 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-token\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-token\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.335826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.335804 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-envoy\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.336036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.336017 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istiod-ca-cert\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.336820 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.336799 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-credential-socket\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.337873 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.337853 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-podinfo\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.338987 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.338967 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-socket\") pod \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\" (UID: \"a58dae4e-ef65-4a85-91ac-35bb3f5db867\") " Oct 31 19:52:42.340143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.332378 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-data" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:42.340143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.335485 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-certs" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:42.340143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.336465 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istiod-ca-cert" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:52:42.340143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.337726 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-credential-socket" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:42.340143 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.339924 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-socket" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:42.342633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.342608 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.343823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.343733 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.344080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.344061 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.344220 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.344205 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.344610 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.344592 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.349950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.349921 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-token" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:42.350074 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.349965 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-kube-api-access-swnnt" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "kube-api-access-swnnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:42.350153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.350085 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-envoy" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:42.352025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.351955 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-podinfo" pod "a58dae4e-ef65-4a85-91ac-35bb3f5db867" (UID: "a58dae4e-ef65-4a85-91ac-35bb3f5db867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:52:42.456633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.454114 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swnnt\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-kube-api-access-swnnt\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.456633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.454155 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.456633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.454174 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.456633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.454189 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58dae4e-ef65-4a85-91ac-35bb3f5db867-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:42.624068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.623908 6481 generic.go:356] "Generic (PLEG): container finished" podID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerID="62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553" exitCode=0 Oct 31 19:52:42.624234 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.624150 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" Oct 31 19:52:42.624234 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.624199 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" event={"ID":"a58dae4e-ef65-4a85-91ac-35bb3f5db867","Type":"ContainerDied","Data":"62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553"} Oct 31 19:52:42.624368 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.624288 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk" event={"ID":"a58dae4e-ef65-4a85-91ac-35bb3f5db867","Type":"ContainerDied","Data":"f7e4dcf0f8af0647580c90ce60c96f7876fd1c98023aadc0a733429cf0e557ea"} Oct 31 19:52:42.624368 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.624320 6481 scope.go:122] "RemoveContainer" containerID="62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553" Oct 31 19:52:42.640166 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.640101 6481 scope.go:122] "RemoveContainer" containerID="62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553" Oct 31 19:52:42.640357 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:52:42.640337 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553\": container with ID starting with 62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553 not found: ID does not exist" containerID="62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553" Oct 31 19:52:42.640426 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.640365 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553"} err="failed to get container status \"62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553\": rpc error: code = NotFound desc = could not find container \"62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553\": container with ID starting with 62db3a93d76d43205fa5a38735de1fea2974974a071b0bc9f773cae24f52c553 not found: ID does not exist" Oct 31 19:52:42.827717 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.827644 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk"] Oct 31 19:52:42.836515 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.836480 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-89hhk"] Oct 31 19:52:42.924306 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:42.924215 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" path="/var/lib/kubelet/pods/a58dae4e-ef65-4a85-91ac-35bb3f5db867/volumes" Oct 31 19:52:43.416024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:43.395765 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:52:44.318017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:44.317024 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 19:52:44.395019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:44.391548 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:52:44.897018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:44.893233 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.926020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:44.923699 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.988022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:44.981441 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:44.988022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:44.982560 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8q6r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:52:44.988022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:44.982602 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:45.037019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:45.035138 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 19:52:45.193075 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:45.189960 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:45.320114 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:45.318973 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:45.330023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:45.320414 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:52:45.448169 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:45.448090 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 19:52:48.727837 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.727798 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr"] Oct 31 19:52:48.728518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.728504 6481 cpu_manager.go:414] "containerMap: removing container" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" Oct 31 19:52:48.728582 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.728520 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" Oct 31 19:52:48.728668 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.728658 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58dae4e-ef65-4a85-91ac-35bb3f5db867" containerName="istio-proxy" Oct 31 19:52:48.734111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.734087 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.767695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.767659 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4jv\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-kube-api-access-9d4jv\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.767922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.767898 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cabd44-169b-40d1-82a5-eb3588801cbe-istiod-ca-cert\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.768026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.768004 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-token\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.768417 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.768387 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-podinfo\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.768530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.768462 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-certs\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.768530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.768498 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-socket\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.768530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.768527 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-data\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.768699 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.768589 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-credential-socket\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.768699 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.768642 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-envoy\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.784427 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.783073 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-872fee-openshift-default-dockercfg-tb9kl\"" Oct 31 19:52:48.832834 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.832802 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr"] Oct 31 19:52:48.873145 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873098 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cabd44-169b-40d1-82a5-eb3588801cbe-istiod-ca-cert\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873335 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873233 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-token\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873335 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873281 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-podinfo\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873463 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873346 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-certs\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873463 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873383 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-socket\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873463 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873419 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-data\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873480 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-credential-socket\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873534 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-envoy\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.873628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.873590 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4jv\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-kube-api-access-9d4jv\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.874947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.874238 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-socket\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.874947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.874336 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-data\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.874947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.874574 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-credential-socket\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.875211 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.875164 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-certs\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.892027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.880463 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-podinfo\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:48.892027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:48.881107 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cabd44-169b-40d1-82a5-eb3588801cbe-istiod-ca-cert\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:49.090068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.089854 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4jv\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-kube-api-access-9d4jv\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:49.111018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.100647 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-envoy\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:49.136020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.128550 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-token\") pod \"e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:49.350288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.350188 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:49.809852 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:52:49.809817 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cabd44_169b_40d1_82a5_eb3588801cbe.slice/crio-7d11917ef462b8c34eba34d8ec7ddcbcfb3b87b5161bcd49d6fce4a1e49d5aa2 WatchSource:0}: Error finding container 7d11917ef462b8c34eba34d8ec7ddcbcfb3b87b5161bcd49d6fce4a1e49d5aa2: Status 404 returned error can't find the container with id 7d11917ef462b8c34eba34d8ec7ddcbcfb3b87b5161bcd49d6fce4a1e49d5aa2 Oct 31 19:52:49.810739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.810714 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr"] Oct 31 19:52:49.813266 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.813233 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:49.813362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.813299 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:49.813362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:49.813332 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:50.212170 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:50.212128 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" event={"ID":"98cabd44-169b-40d1-82a5-eb3588801cbe","Type":"ContainerStarted","Data":"146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e"} Oct 31 19:52:50.212372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:50.212185 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" event={"ID":"98cabd44-169b-40d1-82a5-eb3588801cbe","Type":"ContainerStarted","Data":"7d11917ef462b8c34eba34d8ec7ddcbcfb3b87b5161bcd49d6fce4a1e49d5aa2"} Oct 31 19:52:50.325833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:50.325762 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podStartSLOduration=2.325742125 podStartE2EDuration="2.325742125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:50.324611043 +0000 UTC m=+2141.564834206" watchObservedRunningTime="2026-10-31 19:52:50.325742125 +0000 UTC m=+2141.565965255" Oct 31 19:52:50.351092 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:50.351048 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:50.355274 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:50.355115 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:52:50.355274 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:50.355172 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 19:52:51.354877 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:51.354845 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:51.705147 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:51.705115 6481 patch_prober.go:29] interesting pod/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:52:51.724302 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:51.705190 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:52:52.228127 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.228093 6481 generic.go:356] "Generic (PLEG): container finished" podID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerID="d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3" exitCode=0 Oct 31 19:52:52.228330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.228170 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" event={"ID":"c1d23075-4d02-4dba-83fb-6a7e56a0f31c","Type":"ContainerDied","Data":"d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3"} Oct 31 19:52:52.228490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.228469 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:52.228942 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.228917 6481 scope.go:122] "RemoveContainer" containerID="d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3" Oct 31 19:52:52.229541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.229516 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:52:52.230756 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.230711 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:52.230828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.230796 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:52.230871 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:52.230830 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:52:53.237201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:53.237161 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" event={"ID":"c1d23075-4d02-4dba-83fb-6a7e56a0f31c","Type":"ContainerStarted","Data":"a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070"} Oct 31 19:52:53.253824 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:53.253794 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:52:53.254081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:53.254060 6481 patch_prober.go:29] interesting pod/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:52:53.254185 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:53.254111 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 19:52:54.253820 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:54.253785 6481 patch_prober.go:29] interesting pod/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:52:54.254244 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:54.253836 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 19:52:55.253882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:55.253852 6481 patch_prober.go:29] interesting pod/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:52:55.254403 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:55.253907 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 19:52:56.253450 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:56.253404 6481 patch_prober.go:29] interesting pod/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:52:56.253674 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:56.253465 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 19:52:57.304019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:57.255227 6481 patch_prober.go:29] interesting pod/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:52:57.304019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:57.255290 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 19:52:57.614147 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:52:57.613955 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:52:58.253816 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:58.253776 6481 patch_prober.go:29] interesting pod/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:52:58.254071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:58.253840 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 19:52:59.258448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:59.258376 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:52:59.258867 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:59.258644 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:52:59.259334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:52:59.259313 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:53:21.922330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:21.922253 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j"] Oct 31 19:53:21.926853 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:21.926836 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:21.979396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:21.979361 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j"] Oct 31 19:53:22.021161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.021117 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f37f6f32-serving-cert\"" Oct 31 19:53:22.022040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.022008 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.022174 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.022141 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gd2q\" (UniqueName: \"kubernetes.io/projected/7d7645d3-a774-4624-88dc-2b51b8a0f979-kube-api-access-4gd2q\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.022370 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.022341 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7645d3-a774-4624-88dc-2b51b8a0f979-openshift-service-ca\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.022461 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.022446 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-metrics-certs\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.124497 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.124457 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.124681 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.124517 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gd2q\" (UniqueName: \"kubernetes.io/projected/7d7645d3-a774-4624-88dc-2b51b8a0f979-kube-api-access-4gd2q\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.124681 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.124601 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7645d3-a774-4624-88dc-2b51b8a0f979-openshift-service-ca\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.124681 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:53:22.124609 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-serving-cert: secret "maas-api-e2e-mt-f37f6f32-serving-cert" not found Oct 31 19:53:22.124681 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.124643 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-metrics-certs\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.124840 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:53:22.124732 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls podName:7d7645d3-a774-4624-88dc-2b51b8a0f979 nodeName:}" failed. No retries permitted until 2026-10-31 19:53:22.624710131 +0000 UTC m=+2173.864933256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls") pod "maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" (UID: "7d7645d3-a774-4624-88dc-2b51b8a0f979") : secret "maas-api-e2e-mt-f37f6f32-serving-cert" not found Oct 31 19:53:22.125398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.125368 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7645d3-a774-4624-88dc-2b51b8a0f979-openshift-service-ca\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.127287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.127268 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-metrics-certs\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.178640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.178565 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gd2q\" (UniqueName: \"kubernetes.io/projected/7d7645d3-a774-4624-88dc-2b51b8a0f979-kube-api-access-4gd2q\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.648840 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.648804 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.651611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.651584 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls\") pod \"maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:22.840948 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:22.840917 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:23.227924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:23.224135 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j"] Oct 31 19:53:23.238469 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:53:23.238438 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7645d3_a774_4624_88dc_2b51b8a0f979.slice/crio-32b4936c81e5c876e25e49b831b1760dbdf6daa6cae28eb85412062bb348392e WatchSource:0}: Error finding container 32b4936c81e5c876e25e49b831b1760dbdf6daa6cae28eb85412062bb348392e: Status 404 returned error can't find the container with id 32b4936c81e5c876e25e49b831b1760dbdf6daa6cae28eb85412062bb348392e Oct 31 19:53:23.560654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:23.560558 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" event={"ID":"7d7645d3-a774-4624-88dc-2b51b8a0f979","Type":"ContainerStarted","Data":"32b4936c81e5c876e25e49b831b1760dbdf6daa6cae28eb85412062bb348392e"} Oct 31 19:53:24.572301 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:24.572260 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" event={"ID":"7d7645d3-a774-4624-88dc-2b51b8a0f979","Type":"ContainerStarted","Data":"c62e17383dd720efed4af6d59009727dc01fe9e9504df0eb51137b983bb3f742"} Oct 31 19:53:24.572708 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:24.572362 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:24.690023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:24.689960 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" podStartSLOduration=2.845326405 podStartE2EDuration="3.689946342s" totalImagesPullingTime="844.619937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:24.687820063 +0000 UTC m=+2175.928043194" watchObservedRunningTime="2026-10-31 19:53:24.689946342 +0000 UTC m=+2175.930169472" Oct 31 19:53:27.328770 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:53:27.328733 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:53:30.603505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:30.603475 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:53:51.317703 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.317675 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn"] Oct 31 19:53:51.318139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.317861 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" containerID="cri-o://d85db04ccf435f0f0651f2bf379c16209072b032baad407ee9a3af696fc05552" gracePeriod=30 Oct 31 19:53:51.363379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.363342 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2"] Oct 31 19:53:51.363561 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.363540 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" containerID="cri-o://0526a85e8768b187e3e068f603aa67e6ce4f814641ef8d9889dcd7e3181cf4e7" gracePeriod=30 Oct 31 19:53:51.367565 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.367539 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql"] Oct 31 19:53:51.367733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.367701 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" containerID="cri-o://385cb29c6dbf9464451b17213ea10481418e324893a2aed66b1335bcab0f4f8f" gracePeriod=30 Oct 31 19:53:51.410969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.410930 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x"] Oct 31 19:53:51.411254 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:51.411208 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" containerID="cri-o://06d56de779a5f12504badbf4b23a481f4572b0af34a79f0f477a8efd9ddbaa17" gracePeriod=30 Oct 31 19:53:54.241798 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.241758 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:53:54.242184 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.241829 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:53:54.279137 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.279106 6481 patch_prober.go:29] interesting pod/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:53:54.279275 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.279157 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:53:54.830965 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.830929 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z"] Oct 31 19:53:54.835391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.835374 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:54.930742 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.930711 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-872fee-serving-cert\"" Oct 31 19:53:54.934516 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.934476 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z"] Oct 31 19:53:54.954292 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.954255 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhkpn\" (UniqueName: \"kubernetes.io/projected/618222d1-87d7-4407-ba25-77cc87fa12d4-kube-api-access-xhkpn\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:54.954435 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.954414 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:54.954601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.954577 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618222d1-87d7-4407-ba25-77cc87fa12d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:54.954748 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:54.954730 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.056138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.056062 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhkpn\" (UniqueName: \"kubernetes.io/projected/618222d1-87d7-4407-ba25-77cc87fa12d4-kube-api-access-xhkpn\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.056283 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.056149 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.056283 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.056208 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618222d1-87d7-4407-ba25-77cc87fa12d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.056283 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.056264 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.056443 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:53:55.056420 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-serving-cert: secret "maas-api-e2e-shared-a-w0-872fee-serving-cert" not found Oct 31 19:53:55.056516 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:53:55.056503 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls podName:618222d1-87d7-4407-ba25-77cc87fa12d4 nodeName:}" failed. No retries permitted until 2026-10-31 19:53:55.556482348 +0000 UTC m=+2206.796705469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls") pod "maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" (UID: "618222d1-87d7-4407-ba25-77cc87fa12d4") : secret "maas-api-e2e-shared-a-w0-872fee-serving-cert" not found Oct 31 19:53:55.056975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.056948 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618222d1-87d7-4407-ba25-77cc87fa12d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.058545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.058527 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.140648 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.140526 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhkpn\" (UniqueName: \"kubernetes.io/projected/618222d1-87d7-4407-ba25-77cc87fa12d4-kube-api-access-xhkpn\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.396189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.396099 6481 patch_prober.go:29] interesting pod/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:53:55.396189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.396165 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:53:55.572830 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.572795 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.575307 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.575283 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:55.654583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.654506 6481 generic.go:356] "Generic (PLEG): container finished" podID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerID="5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf" exitCode=0 Oct 31 19:53:55.654730 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.654576 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" event={"ID":"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397","Type":"ContainerDied","Data":"5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf"} Oct 31 19:53:55.655058 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.655043 6481 scope.go:122] "RemoveContainer" containerID="5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf" Oct 31 19:53:55.656685 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.656656 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:53:55.656786 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.656730 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:53:55.656786 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.656782 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:53:55.751274 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:55.751242 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:56.141928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.141900 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z"] Oct 31 19:53:56.143519 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:53:56.143483 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618222d1_87d7_4407_ba25_77cc87fa12d4.slice/crio-189ba34e5d67e71d1899d7b62c0b98dd5babbc33218ac30fa86386cb3c8f43ed WatchSource:0}: Error finding container 189ba34e5d67e71d1899d7b62c0b98dd5babbc33218ac30fa86386cb3c8f43ed: Status 404 returned error can't find the container with id 189ba34e5d67e71d1899d7b62c0b98dd5babbc33218ac30fa86386cb3c8f43ed Oct 31 19:53:56.678073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.665180 6481 generic.go:356] "Generic (PLEG): container finished" podID="6976e490-6df0-42c7-8812-c0950f08d982" containerID="385cb29c6dbf9464451b17213ea10481418e324893a2aed66b1335bcab0f4f8f" exitCode=0 Oct 31 19:53:56.678073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.665276 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" event={"ID":"6976e490-6df0-42c7-8812-c0950f08d982","Type":"ContainerDied","Data":"385cb29c6dbf9464451b17213ea10481418e324893a2aed66b1335bcab0f4f8f"} Oct 31 19:53:56.703051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.702449 6481 generic.go:356] "Generic (PLEG): container finished" podID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerID="06d56de779a5f12504badbf4b23a481f4572b0af34a79f0f477a8efd9ddbaa17" exitCode=0 Oct 31 19:53:56.703051 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.702566 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" event={"ID":"6c5ff706-dd71-48e4-97e9-c010207cc5e3","Type":"ContainerDied","Data":"06d56de779a5f12504badbf4b23a481f4572b0af34a79f0f477a8efd9ddbaa17"} Oct 31 19:53:56.709329 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.709297 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" event={"ID":"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397","Type":"ContainerStarted","Data":"1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4"} Oct 31 19:53:56.711415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.711395 6481 generic.go:356] "Generic (PLEG): container finished" podID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerID="0526a85e8768b187e3e068f603aa67e6ce4f814641ef8d9889dcd7e3181cf4e7" exitCode=0 Oct 31 19:53:56.711580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.711439 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" event={"ID":"d6e80273-92b8-45e1-9b49-63aa625fd4ca","Type":"ContainerDied","Data":"0526a85e8768b187e3e068f603aa67e6ce4f814641ef8d9889dcd7e3181cf4e7"} Oct 31 19:53:56.713139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.713114 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" event={"ID":"618222d1-87d7-4407-ba25-77cc87fa12d4","Type":"ContainerStarted","Data":"189ba34e5d67e71d1899d7b62c0b98dd5babbc33218ac30fa86386cb3c8f43ed"} Oct 31 19:53:56.714851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.714827 6481 generic.go:356] "Generic (PLEG): container finished" podID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerID="d85db04ccf435f0f0651f2bf379c16209072b032baad407ee9a3af696fc05552" exitCode=0 Oct 31 19:53:56.714923 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.714900 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" event={"ID":"78764e47-af66-4b6d-aa36-c47a927b2d3a","Type":"ContainerDied","Data":"d85db04ccf435f0f0651f2bf379c16209072b032baad407ee9a3af696fc05552"} Oct 31 19:53:56.760336 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.760301 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:53:56.910142 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910109 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-certs\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910267 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910203 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-socket\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910267 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910246 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-data\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910389 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910362 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-podinfo\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910442 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910384 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-certs" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:56.910442 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910396 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-token\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-token\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910460 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-envoy\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910488 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-socket" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:56.910957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910530 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-kube-api-access-qxp2k\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910537 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-data" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:56.910957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910590 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-credential-socket\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910616 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78764e47-af66-4b6d-aa36-c47a927b2d3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78764e47-af66-4b6d-aa36-c47a927b2d3a-istiod-ca-cert\") pod \"78764e47-af66-4b6d-aa36-c47a927b2d3a\" (UID: \"78764e47-af66-4b6d-aa36-c47a927b2d3a\") " Oct 31 19:53:56.910957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.910830 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-credential-socket" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:56.911362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.911040 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78764e47-af66-4b6d-aa36-c47a927b2d3a-istiod-ca-cert" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:56.911362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.911310 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:56.911362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.911324 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:56.911362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.911334 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:56.911362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.911343 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78764e47-af66-4b6d-aa36-c47a927b2d3a-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:56.911362 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.911353 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:56.912550 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.912524 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-podinfo" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:53:56.913201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.913053 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-envoy" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:56.913201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.913103 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-kube-api-access-qxp2k" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "kube-api-access-qxp2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:56.913201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.913139 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-token" pod "78764e47-af66-4b6d-aa36-c47a927b2d3a" (UID: "78764e47-af66-4b6d-aa36-c47a927b2d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:56.945733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.945708 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:53:56.949291 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.949270 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:53:56.954114 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:56.954091 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:53:57.012308 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012260 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-envoy\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.012521 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012382 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-credential-socket\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.012521 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012421 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-certs\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.012679 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012581 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-istio-token\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-istio-token\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.012679 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012642 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6976e490-6df0-42c7-8812-c0950f08d982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6976e490-6df0-42c7-8812-c0950f08d982-istiod-ca-cert\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.013520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012689 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-socket\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.013520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012769 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-credential-socket" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.013520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012841 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6976e490-6df0-42c7-8812-c0950f08d982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6976e490-6df0-42c7-8812-c0950f08d982-istio-podinfo\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.013520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012951 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-kube-api-access-5sbxh\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-kube-api-access-5sbxh\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.013520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.012985 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-data\") pod \"6976e490-6df0-42c7-8812-c0950f08d982\" (UID: \"6976e490-6df0-42c7-8812-c0950f08d982\") " Oct 31 19:53:57.013520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.013109 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-socket" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.013520 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.013329 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-data" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.013910 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.013626 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6976e490-6df0-42c7-8812-c0950f08d982-istiod-ca-cert" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:57.013967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.013915 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.013967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.013936 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6976e490-6df0-42c7-8812-c0950f08d982-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.013967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.013951 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.014183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.013966 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.014183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.014067 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.014183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.014084 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78764e47-af66-4b6d-aa36-c47a927b2d3a-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.014183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.014099 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/78764e47-af66-4b6d-aa36-c47a927b2d3a-kube-api-access-qxp2k\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.014183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.014114 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.014430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.014279 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-certs" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.015224 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.015195 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-envoy" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.015628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.015590 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6976e490-6df0-42c7-8812-c0950f08d982-istio-podinfo" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:53:57.017184 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.017160 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-kube-api-access-5sbxh" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "kube-api-access-5sbxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:57.021035 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.020982 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-istio-token" pod "6976e490-6df0-42c7-8812-c0950f08d982" (UID: "6976e490-6df0-42c7-8812-c0950f08d982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:57.135630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.128394 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6976e490-6df0-42c7-8812-c0950f08d982-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.135630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.128439 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbxh\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-kube-api-access-5sbxh\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.135630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.128457 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.135630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.128473 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6976e490-6df0-42c7-8812-c0950f08d982-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.135630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.128490 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6976e490-6df0-42c7-8812-c0950f08d982-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.253177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.236373 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:53:57.256863 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.256827 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-certs\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.260526 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.260500 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-credential-socket\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.260718 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.260699 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istiod-ca-cert\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.260894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.260877 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-podinfo\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.261048 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.261032 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-envoy\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.261183 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.261166 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-socket\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.260305 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-certs" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.260730 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-credential-socket" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.261568 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istiod-ca-cert" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.261615 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-socket" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.261687 6481 patch_prober.go:29] interesting pod/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.262215 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.262314 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-envoy\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.262342 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-data\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.262609 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-data" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.263384 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.262911 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-kube-api-access-hd5lq\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-kube-api-access-hd5lq\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.264623 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.264602 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-token\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-token\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.264767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.264751 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-certs\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.265055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.265039 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-podinfo\") pod \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\" (UID: \"6c5ff706-dd71-48e4-97e9-c010207cc5e3\") " Oct 31 19:53:57.265201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.265187 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-kube-api-access-mzc68\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-kube-api-access-mzc68\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.265304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.265292 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-data\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.269180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.269153 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istiod-ca-cert\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.269400 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.269384 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-token\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-token\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.269582 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.269566 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-credential-socket\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.269698 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.269684 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-socket\") pod \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\" (UID: \"d6e80273-92b8-45e1-9b49-63aa625fd4ca\") " Oct 31 19:53:57.287061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.270666 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.287622 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.287595 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.287779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.287766 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.287920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.287899 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.288054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.288038 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.288164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.276206 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-data" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.288257 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.276378 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-certs" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.288347 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.278421 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-envoy" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.288443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.280640 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-socket" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.288549 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.280830 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-credential-socket" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.288653 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.280886 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-envoy" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:57.288737 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.284902 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-kube-api-access-mzc68" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "kube-api-access-mzc68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:57.288821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.285226 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istiod-ca-cert" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:57.288912 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.287492 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-token" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:57.299436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.289827 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-kube-api-access-hd5lq" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "kube-api-access-hd5lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:57.299436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.294073 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-podinfo" pod "6c5ff706-dd71-48e4-97e9-c010207cc5e3" (UID: "6c5ff706-dd71-48e4-97e9-c010207cc5e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:53:57.299436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.294188 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-token" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:57.299436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.295423 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-podinfo" pod "d6e80273-92b8-45e1-9b49-63aa625fd4ca" (UID: "d6e80273-92b8-45e1-9b49-63aa625fd4ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:53:57.343732 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:53:57.343536 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391598 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391636 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391652 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391668 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391683 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391697 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5lq\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-kube-api-access-hd5lq\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391713 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391727 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c5ff706-dd71-48e4-97e9-c010207cc5e3-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391742 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c5ff706-dd71-48e4-97e9-c010207cc5e3-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391756 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzc68\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-kube-api-access-mzc68\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391769 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391783 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.395109 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.391797 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e80273-92b8-45e1-9b49-63aa625fd4ca-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:57.722714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.722674 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" Oct 31 19:53:57.722714 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.722706 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn" event={"ID":"78764e47-af66-4b6d-aa36-c47a927b2d3a","Type":"ContainerDied","Data":"7d1975db037e7ccb4e04160ce6f91248ffcb3d43e9ba06ce316eb6f866a290fe"} Oct 31 19:53:57.723264 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.722772 6481 scope.go:122] "RemoveContainer" containerID="d85db04ccf435f0f0651f2bf379c16209072b032baad407ee9a3af696fc05552" Oct 31 19:53:57.724272 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.724247 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" Oct 31 19:53:57.724408 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.724255 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql" event={"ID":"6976e490-6df0-42c7-8812-c0950f08d982","Type":"ContainerDied","Data":"d41890ecee228bbc252ba7f64525e247015562026a8fe915d5257ea6e869c80b"} Oct 31 19:53:57.725809 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.725778 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" Oct 31 19:53:57.725922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.725829 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x" event={"ID":"6c5ff706-dd71-48e4-97e9-c010207cc5e3","Type":"ContainerDied","Data":"2ca5c8a7b8d62d5144960e9341cdf7dc86e33c72b681e28d8a61b836e953c502"} Oct 31 19:53:57.730556 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.730527 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" event={"ID":"d6e80273-92b8-45e1-9b49-63aa625fd4ca","Type":"ContainerDied","Data":"f485923cc1bb3593020f3f018ff3b60dae08ed99af06685633c92aa69d12acd0"} Oct 31 19:53:57.730663 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.730531 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2" Oct 31 19:53:57.734205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.734123 6481 scope.go:122] "RemoveContainer" containerID="385cb29c6dbf9464451b17213ea10481418e324893a2aed66b1335bcab0f4f8f" Oct 31 19:53:57.735669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.735642 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" event={"ID":"618222d1-87d7-4407-ba25-77cc87fa12d4","Type":"ContainerStarted","Data":"9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca"} Oct 31 19:53:57.735798 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.735784 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:53:57.745130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.745110 6481 scope.go:122] "RemoveContainer" containerID="06d56de779a5f12504badbf4b23a481f4572b0af34a79f0f477a8efd9ddbaa17" Oct 31 19:53:57.755910 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.755730 6481 scope.go:122] "RemoveContainer" containerID="0526a85e8768b187e3e068f603aa67e6ce4f814641ef8d9889dcd7e3181cf4e7" Oct 31 19:53:57.894804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.894769 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn"] Oct 31 19:53:57.898274 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:57.898245 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-965nn"] Oct 31 19:53:58.073133 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.071235 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" podStartSLOduration=2.956661755 podStartE2EDuration="4.071213015s" totalImagesPullingTime="1.11455126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:58.02679931 +0000 UTC m=+2209.267022486" watchObservedRunningTime="2026-10-31 19:53:58.071213015 +0000 UTC m=+2209.311436146" Oct 31 19:53:58.199477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.199446 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2"] Oct 31 19:53:58.213579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.213545 6481 patch_prober.go:29] interesting pod/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:53:58.213842 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.213816 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 19:53:58.214511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.214489 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-4n2m2"] Oct 31 19:53:58.403153 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.403052 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql"] Oct 31 19:53:58.411966 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.411930 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-v8wql"] Oct 31 19:53:58.415266 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.415234 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x"] Oct 31 19:53:58.418240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.418211 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b5155c-openshift-default-68b4448597-2vb4x"] Oct 31 19:53:58.900885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.900852 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6976e490-6df0-42c7-8812-c0950f08d982" path="/var/lib/kubelet/pods/6976e490-6df0-42c7-8812-c0950f08d982/volumes" Oct 31 19:53:58.901438 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.901421 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" path="/var/lib/kubelet/pods/6c5ff706-dd71-48e4-97e9-c010207cc5e3/volumes" Oct 31 19:53:58.901891 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.901879 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" path="/var/lib/kubelet/pods/78764e47-af66-4b6d-aa36-c47a927b2d3a/volumes" Oct 31 19:53:58.902680 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:58.902659 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" path="/var/lib/kubelet/pods/d6e80273-92b8-45e1-9b49-63aa625fd4ca/volumes" Oct 31 19:53:59.206669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:59.206574 6481 patch_prober.go:29] interesting pod/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:53:59.206669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:53:59.206643 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 19:54:00.206349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:00.206315 6481 patch_prober.go:29] interesting pod/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:54:00.206715 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:00.206377 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 19:54:01.206360 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:01.206322 6481 patch_prober.go:29] interesting pod/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:54:01.206882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:01.206380 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 19:54:02.279880 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:02.279840 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:54:02.280644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:02.280320 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:54:02.282902 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:02.282880 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:54:03.760541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:03.760508 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 19:54:07.506022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.505974 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb"] Oct 31 19:54:07.506431 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.506241 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" containerID="cri-o://f44888c99d5e8ac8b69dc3e57a5cf3329c7da96885a22b826edacb148d252feb" gracePeriod=30 Oct 31 19:54:07.551893 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.551855 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s"] Oct 31 19:54:07.552159 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.552119 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" containerID="cri-o://82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7" gracePeriod=30 Oct 31 19:54:07.556304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.556272 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8"] Oct 31 19:54:07.556545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.556499 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" containerID="cri-o://259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4" gracePeriod=30 Oct 31 19:54:07.596976 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.596943 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr"] Oct 31 19:54:07.718817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.718783 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh"] Oct 31 19:54:07.719080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.719040 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" containerID="cri-o://d8cef71e5fd12d6a073b59309bf3f0d458070ffc2c65324c1d55704d51bb2898" gracePeriod=30 Oct 31 19:54:07.722065 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.722040 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj"] Oct 31 19:54:07.722259 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.722217 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" containerID="cri-o://5196df3fdd4bc33f1edd8bd5b775673f5e192e39fa6d7e9d876a0267d430af3c" gracePeriod=30 Oct 31 19:54:07.851821 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:07.851732 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" containerID="cri-o://16b7e76caa73ed9765c8e3d7a6dafa8bbc02d80cc5dd43189957d402e5315188" gracePeriod=30 Oct 31 19:54:09.446596 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:09.446557 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9"] Oct 31 19:54:09.447126 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:09.446761 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" podUID="a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" containerName="maas-api" containerID="cri-o://21050ce929ea535ce699f0437774fe18b818bd5774e6844ae76517b2c5185b40" gracePeriod=30 Oct 31 19:54:09.941979 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:09.941937 6481 generic.go:356] "Generic (PLEG): container finished" podID="a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" containerID="21050ce929ea535ce699f0437774fe18b818bd5774e6844ae76517b2c5185b40" exitCode=0 Oct 31 19:54:09.942547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:09.942516 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" event={"ID":"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3","Type":"ContainerDied","Data":"21050ce929ea535ce699f0437774fe18b818bd5774e6844ae76517b2c5185b40"} Oct 31 19:54:09.942767 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:09.942750 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" event={"ID":"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3","Type":"ContainerDied","Data":"705c9f798b245b7d11de3a1d2d17d561093847d1dac3811e622febb7cfc5e64c"} Oct 31 19:54:09.942875 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:09.942863 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705c9f798b245b7d11de3a1d2d17d561093847d1dac3811e622febb7cfc5e64c" Oct 31 19:54:10.060055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.057248 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:54:10.115398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.115320 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-openshift-service-ca\") pod \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " Oct 31 19:54:10.115398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.115373 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-maas-api-tls\") pod \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " Oct 31 19:54:10.115579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.115453 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-metrics-certs\") pod \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " Oct 31 19:54:10.115666 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.115648 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-kube-api-access-vd8tv\" (UniqueName: \"kubernetes.io/projected/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-kube-api-access-vd8tv\") pod \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\" (UID: \"a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3\") " Oct 31 19:54:10.115770 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.115744 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-openshift-service-ca" pod "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" (UID: "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:10.116285 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.116262 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:10.117753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.117727 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-kube-api-access-vd8tv" pod "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" (UID: "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3"). InnerVolumeSpecName "kube-api-access-vd8tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:10.117851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.117753 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-metrics-certs" pod "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" (UID: "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:54:10.117851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.117797 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-maas-api-tls" pod "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" (UID: "a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:54:10.217817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.217782 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8tv\" (UniqueName: \"kubernetes.io/projected/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-kube-api-access-vd8tv\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:10.217817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.217815 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:10.217817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.217824 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:10.949205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:10.949165 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9" Oct 31 19:54:11.107695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:11.107661 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9"] Oct 31 19:54:11.153645 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:11.153573 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-abb7688a-d8db8758c-9rxq9"] Oct 31 19:54:12.950029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.947873 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" path="/var/lib/kubelet/pods/a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3/volumes" Oct 31 19:54:12.950029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.948590 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:54:12.981086 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.979213 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:54:12.984490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.984376 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-envoy\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.984490 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.984443 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-socket\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.984987 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.984607 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-kube-api-access-mhsc8\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-kube-api-access-mhsc8\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.984987 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.984923 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-credential-socket\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.985421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.985132 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-podinfo\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.985421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.985270 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-token\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-token\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.985421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.985366 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istiod-ca-cert\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.985963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.985909 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-certs\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.986173 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.986102 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-data\") pod \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\" (UID: \"71c4c87e-9ef7-44a0-b61c-7939a1a40205\") " Oct 31 19:54:12.991009 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:12.987303 6481 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9733ccf_04a4_4b33_913e_143456ef98a0.slice/crio-16b7e76caa73ed9765c8e3d7a6dafa8bbc02d80cc5dd43189957d402e5315188.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9733ccf_04a4_4b33_913e_143456ef98a0.slice/crio-conmon-16b7e76caa73ed9765c8e3d7a6dafa8bbc02d80cc5dd43189957d402e5315188.scope\": RecentStats: unable to find data in memory cache]" Oct 31 19:54:12.991009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.988202 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-socket" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.991009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.988533 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-certs" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.991009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.988773 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-data" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.991009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.989303 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istiod-ca-cert" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:12.991009 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.990941 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-credential-socket" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:12.994445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.994300 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-token" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:12.995330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.995203 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-podinfo" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:12.995330 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:12.995291 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-envoy" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.089079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.084886 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.094957 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-certs\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095047 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b383e843-8685-4714-b21a-ded1019edf5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b383e843-8685-4714-b21a-ded1019edf5b-istiod-ca-cert\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095109 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b383e843-8685-4714-b21a-ded1019edf5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b383e843-8685-4714-b21a-ded1019edf5b-istio-podinfo\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095153 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-envoy\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095213 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-credential-socket\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095317 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-certs" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095325 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-data\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095478 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-socket\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095585 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-kube-api-access-l6xvj\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-kube-api-access-l6xvj\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.095702 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-istio-token\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-istio-token\") pod \"b383e843-8685-4714-b21a-ded1019edf5b\" (UID: \"b383e843-8685-4714-b21a-ded1019edf5b\") " Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096513 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096537 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096550 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096565 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096578 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096592 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c4c87e-9ef7-44a0-b61c-7939a1a40205-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096608 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096622 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.096950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.096636 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c4c87e-9ef7-44a0-b61c-7939a1a40205-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.107073 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.105253 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-data" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.111829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.110279 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-credential-socket" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.111829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.110828 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b383e843-8685-4714-b21a-ded1019edf5b-istiod-ca-cert" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:13.112354 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.112318 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-socket" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.126595 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.126089 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b383e843-8685-4714-b21a-ded1019edf5b-istio-podinfo" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.205715 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-credential-socket\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.205879 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-kube-api-access-bmwbg\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-kube-api-access-bmwbg\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.205935 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-podinfo\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.206188 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e04162-c773-4620-90c1-f8eb6f659d1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e04162-c773-4620-90c1-f8eb6f659d1f-istiod-ca-cert\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.206293 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-token\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-token\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.206330 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-socket\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.206361 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-certs\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.206416 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-envoy\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.265482 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.206478 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-data\") pod \"c8e04162-c773-4620-90c1-f8eb6f659d1f\" (UID: \"c8e04162-c773-4620-90c1-f8eb6f659d1f\") " Oct 31 19:54:13.269097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.267111 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b383e843-8685-4714-b21a-ded1019edf5b-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.269097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.267150 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b383e843-8685-4714-b21a-ded1019edf5b-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.269097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.267165 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.269097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.267179 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.269097 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.267195 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.303879 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.269616 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-credential-socket" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.304287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.304251 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-kube-api-access-mhsc8" pod "71c4c87e-9ef7-44a0-b61c-7939a1a40205" (UID: "71c4c87e-9ef7-44a0-b61c-7939a1a40205"). InnerVolumeSpecName "kube-api-access-mhsc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.305280 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.304961 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-socket" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.310021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.305617 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-certs" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.321055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.313201 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e04162-c773-4620-90c1-f8eb6f659d1f-istiod-ca-cert" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:13.339182 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.339093 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-data" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.368867 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.368709 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e04162-c773-4620-90c1-f8eb6f659d1f-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.369342 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.369323 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.369498 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.369484 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.369811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.369798 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.370036 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.370021 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsc8\" (UniqueName: \"kubernetes.io/projected/71c4c87e-9ef7-44a0-b61c-7939a1a40205-kube-api-access-mhsc8\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.370154 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.370141 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.370261 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.369696 6481 generic.go:356] "Generic (PLEG): container finished" podID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerID="82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7" exitCode=0 Oct 31 19:54:13.374452 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.369726 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" event={"ID":"71c4c87e-9ef7-44a0-b61c-7939a1a40205","Type":"ContainerDied","Data":"82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7"} Oct 31 19:54:13.374452 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.370457 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" event={"ID":"71c4c87e-9ef7-44a0-b61c-7939a1a40205","Type":"ContainerDied","Data":"abac1fd65530abf199034447d25e94ece9adea07d606d24a7bda5084fcec0355"} Oct 31 19:54:13.374452 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.370492 6481 scope.go:122] "RemoveContainer" containerID="82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7" Oct 31 19:54:13.374452 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.369771 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s" Oct 31 19:54:13.385924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.385891 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-envoy" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.386080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.386032 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-token" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.386701 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.386649 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-podinfo" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:13.392774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.387789 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-envoy" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.392774 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.388217 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-istio-token" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.411182 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.406527 6481 scope.go:122] "RemoveContainer" containerID="82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7" Oct 31 19:54:13.411182 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:13.407345 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7\": container with ID starting with 82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7 not found: ID does not exist" containerID="82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7" Oct 31 19:54:13.411182 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.407383 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7"} err="failed to get container status \"82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7\": rpc error: code = NotFound desc = could not find container \"82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7\": container with ID starting with 82671f0ee42b0a71e023e37ce54354508cab43fe2e18b9830a80904c874bfae7 not found: ID does not exist" Oct 31 19:54:13.415040 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.413784 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-kube-api-access-bmwbg" pod "c8e04162-c773-4620-90c1-f8eb6f659d1f" (UID: "c8e04162-c773-4620-90c1-f8eb6f659d1f"). InnerVolumeSpecName "kube-api-access-bmwbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.424228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.424192 6481 generic.go:356] "Generic (PLEG): container finished" podID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerID="5196df3fdd4bc33f1edd8bd5b775673f5e192e39fa6d7e9d876a0267d430af3c" exitCode=0 Oct 31 19:54:13.424653 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.424621 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" event={"ID":"23229f4b-ec12-465d-ac48-1e7be99a337d","Type":"ContainerDied","Data":"5196df3fdd4bc33f1edd8bd5b775673f5e192e39fa6d7e9d876a0267d430af3c"} Oct 31 19:54:13.426881 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.426853 6481 generic.go:356] "Generic (PLEG): container finished" podID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerID="f44888c99d5e8ac8b69dc3e57a5cf3329c7da96885a22b826edacb148d252feb" exitCode=0 Oct 31 19:54:13.427139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.427115 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" event={"ID":"c8e04162-c773-4620-90c1-f8eb6f659d1f","Type":"ContainerDied","Data":"f44888c99d5e8ac8b69dc3e57a5cf3329c7da96885a22b826edacb148d252feb"} Oct 31 19:54:13.427298 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.427282 6481 scope.go:122] "RemoveContainer" containerID="f44888c99d5e8ac8b69dc3e57a5cf3329c7da96885a22b826edacb148d252feb" Oct 31 19:54:13.427564 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.427548 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb" Oct 31 19:54:13.438665 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.438464 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-kube-api-access-l6xvj" pod "b383e843-8685-4714-b21a-ded1019edf5b" (UID: "b383e843-8685-4714-b21a-ded1019edf5b"). InnerVolumeSpecName "kube-api-access-l6xvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.445786 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.445534 6481 generic.go:356] "Generic (PLEG): container finished" podID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerID="16b7e76caa73ed9765c8e3d7a6dafa8bbc02d80cc5dd43189957d402e5315188" exitCode=0 Oct 31 19:54:13.445786 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.445714 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" event={"ID":"f9733ccf-04a4-4b33-913e-143456ef98a0","Type":"ContainerDied","Data":"16b7e76caa73ed9765c8e3d7a6dafa8bbc02d80cc5dd43189957d402e5315188"} Oct 31 19:54:13.451188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.451156 6481 generic.go:356] "Generic (PLEG): container finished" podID="b383e843-8685-4714-b21a-ded1019edf5b" containerID="259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4" exitCode=0 Oct 31 19:54:13.451466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.451446 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" event={"ID":"b383e843-8685-4714-b21a-ded1019edf5b","Type":"ContainerDied","Data":"259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4"} Oct 31 19:54:13.451592 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.451579 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" event={"ID":"b383e843-8685-4714-b21a-ded1019edf5b","Type":"ContainerDied","Data":"72f76978964638a8e095bdb5b2b7f0a473ded5c78a7fcd00657e5a39e30a8498"} Oct 31 19:54:13.451807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.451793 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8" Oct 31 19:54:13.471318 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.471290 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:54:13.476547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.473251 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.476547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.473284 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.476547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.473313 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmwbg\" (UniqueName: \"kubernetes.io/projected/c8e04162-c773-4620-90c1-f8eb6f659d1f-kube-api-access-bmwbg\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.476547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.473329 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e04162-c773-4620-90c1-f8eb6f659d1f-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.476547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.473344 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6xvj\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-kube-api-access-l6xvj\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.476547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.473357 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b383e843-8685-4714-b21a-ded1019edf5b-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.476547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.473378 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b383e843-8685-4714-b21a-ded1019edf5b-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.478394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.478322 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f"] Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480100 6481 cpu_manager.go:414] "containerMap: removing container" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480124 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480160 6481 cpu_manager.go:414] "containerMap: removing container" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480171 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480193 6481 cpu_manager.go:414] "containerMap: removing container" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480201 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480238 6481 cpu_manager.go:414] "containerMap: removing container" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480246 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480263 6481 cpu_manager.go:414] "containerMap: removing container" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480272 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480289 6481 cpu_manager.go:414] "containerMap: removing container" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480296 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480359 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480367 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480421 6481 cpu_manager.go:414] "containerMap: removing container" podUID="a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" containerName="maas-api" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480429 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" containerName="maas-api" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480457 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480465 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480786 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5ff706-dd71-48e4-97e9-c010207cc5e3" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480822 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6976e490-6df0-42c7-8812-c0950f08d982" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480832 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480852 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480863 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="78764e47-af66-4b6d-aa36-c47a927b2d3a" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480872 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480893 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e80273-92b8-45e1-9b49-63aa625fd4ca" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480936 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f5b6c1-f081-4a89-94a6-a6b82ab64cf3" containerName="maas-api" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.480954 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" containerName="istio-proxy" Oct 31 19:54:13.488833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.486748 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.496938 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:13.496211 6481 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e04162_c773_4620_90c1_f8eb6f659d1f.slice/crio-6acea23f25df4ed1fe415f30e2c40c7d59813e7f322432509a4be157007ae361\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e04162_c773_4620_90c1_f8eb6f659d1f.slice\": RecentStats: unable to find data in memory cache]" Oct 31 19:54:13.509932 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.508862 6481 scope.go:122] "RemoveContainer" containerID="259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4" Oct 31 19:54:13.511480 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.511353 6481 generic.go:356] "Generic (PLEG): container finished" podID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerID="d8cef71e5fd12d6a073b59309bf3f0d458070ffc2c65324c1d55704d51bb2898" exitCode=0 Oct 31 19:54:13.511480 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.511399 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" event={"ID":"2bb6cda9-dd33-4a69-a1b2-8341a79026af","Type":"ContainerDied","Data":"d8cef71e5fd12d6a073b59309bf3f0d458070ffc2c65324c1d55704d51bb2898"} Oct 31 19:54:13.529458 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.529432 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f"] Oct 31 19:54:13.534429 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.534310 6481 scope.go:122] "RemoveContainer" containerID="7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213" Oct 31 19:54:13.535927 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.535836 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-a5ba62-openshift-default-dockercfg-t49zk\"" Oct 31 19:54:13.575069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575037 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-token\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-token\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575099 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-credential-socket\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575145 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-podinfo\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575180 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-kube-api-access-84qkd\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-kube-api-access-84qkd\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575202 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23229f4b-ec12-465d-ac48-1e7be99a337d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23229f4b-ec12-465d-ac48-1e7be99a337d-istiod-ca-cert\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575226 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-certs\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575281 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-data\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575350 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-socket\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575515 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-envoy\") pod \"23229f4b-ec12-465d-ac48-1e7be99a337d\" (UID: \"23229f4b-ec12-465d-ac48-1e7be99a337d\") " Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575766 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-credential-socket" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575780 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-data" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575823 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-istio-token\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575836 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23229f4b-ec12-465d-ac48-1e7be99a337d-istiod-ca-cert" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575859 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-certs\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575932 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd23d352-acb4-4b63-addf-478cb67bc561-istiod-ca-cert\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575957 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-data\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.575982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.575972 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-socket" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.576455 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576122 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-certs" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.576455 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576136 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-envoy\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.576455 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576213 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd23d352-acb4-4b63-addf-478cb67bc561-istio-podinfo\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.576455 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576294 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-socket\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.576584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576486 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zz7\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-kube-api-access-r6zz7\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.576643 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576622 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-credential-socket\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.576872 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576848 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.576943 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576879 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.576943 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576896 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.576943 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576912 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23229f4b-ec12-465d-ac48-1e7be99a337d-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.576943 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.576925 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.577761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.577734 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-token" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.577964 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.577934 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-podinfo" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:13.578078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.578042 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-envoy" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.578527 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.578512 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-kube-api-access-84qkd" pod "23229f4b-ec12-465d-ac48-1e7be99a337d" (UID: "23229f4b-ec12-465d-ac48-1e7be99a337d"). InnerVolumeSpecName "kube-api-access-84qkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.601960 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.601935 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:54:13.602080 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.602058 6481 scope.go:122] "RemoveContainer" containerID="259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4" Oct 31 19:54:13.609542 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:13.609091 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4\": container with ID starting with 259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4 not found: ID does not exist" containerID="259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4" Oct 31 19:54:13.609542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.609141 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4"} err="failed to get container status \"259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4\": rpc error: code = NotFound desc = could not find container \"259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4\": container with ID starting with 259f0ac0456f864dab1fc68055a04fbc3438bd7fe418aab94ffed6c9a79451b4 not found: ID does not exist" Oct 31 19:54:13.609542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.609167 6481 scope.go:122] "RemoveContainer" containerID="7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213" Oct 31 19:54:13.609542 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:13.609444 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213\": container with ID starting with 7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213 not found: ID does not exist" containerID="7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213" Oct 31 19:54:13.609542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.609473 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213"} err="failed to get container status \"7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213\": rpc error: code = NotFound desc = could not find container \"7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213\": container with ID starting with 7238d0430f73309daf538c25a94908bd9905a2f202e7a08906a71781821cb213 not found: ID does not exist" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.637208 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.688780 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-certs\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.688840 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-podinfo\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.688900 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-data\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.688957 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-token\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-token\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.689019 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istiod-ca-cert\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.689098 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-certs\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.689130 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-socket\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.689206 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-credential-socket\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.695881 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-envoy\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.695953 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-podinfo\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696058 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-kube-api-access-8bb78\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-kube-api-access-8bb78\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696099 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-socket\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696160 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-envoy\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696204 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-credential-socket\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696253 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-data\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696311 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-kube-api-access-gxx25\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-kube-api-access-gxx25\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696358 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9733ccf-04a4-4b33-913e-143456ef98a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9733ccf-04a4-4b33-913e-143456ef98a0-istiod-ca-cert\") pod \"f9733ccf-04a4-4b33-913e-143456ef98a0\" (UID: \"f9733ccf-04a4-4b33-913e-143456ef98a0\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696392 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-token\") pod \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\" (UID: \"2bb6cda9-dd33-4a69-a1b2-8341a79026af\") " Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696670 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zz7\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-kube-api-access-r6zz7\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696808 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-credential-socket\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.696975 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-istio-token\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.697021 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-certs\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.697054 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd23d352-acb4-4b63-addf-478cb67bc561-istiod-ca-cert\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.697077 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-data\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.697167 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-envoy\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.697252 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd23d352-acb4-4b63-addf-478cb67bc561-istio-podinfo\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.697296 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-socket\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.699092 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-credential-socket\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.699358 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-credential-socket" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.699552 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-data" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.700087 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-certs" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.700299 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-certs" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.717431 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-data" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720200 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-envoy" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720435 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-socket" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720611 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-certs\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720822 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720840 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720855 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720869 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720885 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23229f4b-ec12-465d-ac48-1e7be99a337d-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720899 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84qkd\" (UniqueName: \"kubernetes.io/projected/23229f4b-ec12-465d-ac48-1e7be99a337d-kube-api-access-84qkd\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720914 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720929 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720943 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720958 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720971 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.721571 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd23d352-acb4-4b63-addf-478cb67bc561-istiod-ca-cert\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.721791 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-data\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.728334 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-socket\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.720615 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-credential-socket" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.724873 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-token" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.725208 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istiod-ca-cert" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.725734 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-socket" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.730858 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-kube-api-access-8bb78" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "kube-api-access-8bb78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.731110 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-envoy\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.731817 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9733ccf-04a4-4b33-913e-143456ef98a0-istiod-ca-cert" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.734510 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-podinfo" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.734899 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-podinfo" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:13.756188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.748571 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd23d352-acb4-4b63-addf-478cb67bc561-istio-podinfo\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.764635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.763690 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-kube-api-access-gxx25" pod "f9733ccf-04a4-4b33-913e-143456ef98a0" (UID: "f9733ccf-04a4-4b33-913e-143456ef98a0"). InnerVolumeSpecName "kube-api-access-gxx25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.764635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.764548 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-token" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:13.779320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.778172 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s"] Oct 31 19:54:13.788483 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.788439 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-tm95s"] Oct 31 19:54:13.794112 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.794032 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zz7\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-kube-api-access-r6zz7\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.797971 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.797935 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-envoy" pod "2bb6cda9-dd33-4a69-a1b2-8341a79026af" (UID: "2bb6cda9-dd33-4a69-a1b2-8341a79026af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:13.799732 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.799524 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-istio-token\") pod \"e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828075 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828115 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bb78\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-kube-api-access-8bb78\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828131 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828146 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828160 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxx25\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-kube-api-access-gxx25\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828173 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9733ccf-04a4-4b33-913e-143456ef98a0-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828185 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828198 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828211 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9733ccf-04a4-4b33-913e-143456ef98a0-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828224 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb6cda9-dd33-4a69-a1b2-8341a79026af-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.828552 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.828237 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9733ccf-04a4-4b33-913e-143456ef98a0-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:13.884609 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.884524 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:13.919285 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.919237 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb"] Oct 31 19:54:13.926676 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:13.926639 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-ttxwb"] Oct 31 19:54:14.284811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.284764 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8"] Oct 31 19:54:14.285317 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.284835 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-k56f8"] Oct 31 19:54:14.320841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.320798 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f"] Oct 31 19:54:14.322839 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:14.322810 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd23d352_acb4_4b63_addf_478cb67bc561.slice/crio-1307a8fcb6e8837c5873bd635671ef3194c1d5432143d62530996d416bf85b19 WatchSource:0}: Error finding container 1307a8fcb6e8837c5873bd635671ef3194c1d5432143d62530996d416bf85b19: Status 404 returned error can't find the container with id 1307a8fcb6e8837c5873bd635671ef3194c1d5432143d62530996d416bf85b19 Oct 31 19:54:14.325621 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.325592 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:54:14.325689 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.325658 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:54:14.325722 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.325688 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:54:14.521177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.521145 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" event={"ID":"2bb6cda9-dd33-4a69-a1b2-8341a79026af","Type":"ContainerDied","Data":"476325458acc792a486ec0e646d3c150d0e45eb6cb4474eabace97255ce911be"} Oct 31 19:54:14.521357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.521149 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh" Oct 31 19:54:14.521357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.521199 6481 scope.go:122] "RemoveContainer" containerID="d8cef71e5fd12d6a073b59309bf3f0d458070ffc2c65324c1d55704d51bb2898" Oct 31 19:54:14.526039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.524148 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" Oct 31 19:54:14.526039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.524184 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj" event={"ID":"23229f4b-ec12-465d-ac48-1e7be99a337d","Type":"ContainerDied","Data":"9762ff7506e01670e293e74626bb34bf05b5e4beb376ccedba3d3f53239c1f31"} Oct 31 19:54:14.528989 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.528965 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" event={"ID":"dd23d352-acb4-4b63-addf-478cb67bc561","Type":"ContainerStarted","Data":"9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888"} Oct 31 19:54:14.529123 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.529023 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" event={"ID":"dd23d352-acb4-4b63-addf-478cb67bc561","Type":"ContainerStarted","Data":"1307a8fcb6e8837c5873bd635671ef3194c1d5432143d62530996d416bf85b19"} Oct 31 19:54:14.531558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.531539 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" event={"ID":"f9733ccf-04a4-4b33-913e-143456ef98a0","Type":"ContainerDied","Data":"227f806f86fc656e3100acff2e69f3957101b763e3381aa1fb53100abb3bf999"} Oct 31 19:54:14.531651 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.531580 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr" Oct 31 19:54:14.548820 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.548802 6481 scope.go:122] "RemoveContainer" containerID="5196df3fdd4bc33f1edd8bd5b775673f5e192e39fa6d7e9d876a0267d430af3c" Oct 31 19:54:14.558563 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.558545 6481 scope.go:122] "RemoveContainer" containerID="16b7e76caa73ed9765c8e3d7a6dafa8bbc02d80cc5dd43189957d402e5315188" Oct 31 19:54:14.675373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.675344 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj"] Oct 31 19:54:14.684874 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.684849 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-fgttj"] Oct 31 19:54:14.786827 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.786796 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr"] Oct 31 19:54:14.791933 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.791862 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-967dee-openshift-default-564986b997-8kkxr"] Oct 31 19:54:14.880773 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.880741 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23229f4b-ec12-465d-ac48-1e7be99a337d" path="/var/lib/kubelet/pods/23229f4b-ec12-465d-ac48-1e7be99a337d/volumes" Oct 31 19:54:14.881252 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.881238 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c4c87e-9ef7-44a0-b61c-7939a1a40205" path="/var/lib/kubelet/pods/71c4c87e-9ef7-44a0-b61c-7939a1a40205/volumes" Oct 31 19:54:14.881687 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.881674 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b383e843-8685-4714-b21a-ded1019edf5b" path="/var/lib/kubelet/pods/b383e843-8685-4714-b21a-ded1019edf5b/volumes" Oct 31 19:54:14.882450 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.882431 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e04162-c773-4620-90c1-f8eb6f659d1f" path="/var/lib/kubelet/pods/c8e04162-c773-4620-90c1-f8eb6f659d1f/volumes" Oct 31 19:54:14.882882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.882870 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" path="/var/lib/kubelet/pods/f9733ccf-04a4-4b33-913e-143456ef98a0/volumes" Oct 31 19:54:14.885262 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.885247 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:14.892071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.892051 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh"] Oct 31 19:54:14.898285 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:14.898265 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-52vbh"] Oct 31 19:54:15.005160 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:15.005088 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podStartSLOduration=2.005073437 podStartE2EDuration="2.005073437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:15.00268679 +0000 UTC m=+2226.242909923" watchObservedRunningTime="2026-10-31 19:54:15.005073437 +0000 UTC m=+2226.245296622" Oct 31 19:54:15.890628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:15.890599 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:16.097099 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.097065 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc"] Oct 31 19:54:16.097292 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.097268 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" podUID="7ff3d4c5-512f-4be0-b799-2d2d1527bdee" containerName="maas-api" containerID="cri-o://e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400" gracePeriod=30 Oct 31 19:54:16.475736 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.475659 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:54:16.578088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.562980 6481 generic.go:356] "Generic (PLEG): container finished" podID="7ff3d4c5-512f-4be0-b799-2d2d1527bdee" containerID="e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400" exitCode=0 Oct 31 19:54:16.578088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.564360 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" Oct 31 19:54:16.578088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.564583 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" event={"ID":"7ff3d4c5-512f-4be0-b799-2d2d1527bdee","Type":"ContainerDied","Data":"e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400"} Oct 31 19:54:16.578088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.564667 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc" event={"ID":"7ff3d4c5-512f-4be0-b799-2d2d1527bdee","Type":"ContainerDied","Data":"bfb456753c80ce247ac49fa6b75564dcbad9399bad8ca773e3105d550c7dd16e"} Oct 31 19:54:16.578088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.564697 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:16.578088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.564727 6481 scope.go:122] "RemoveContainer" containerID="e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400" Oct 31 19:54:16.580024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.579120 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-kube-api-access-b6ntd\" (UniqueName: \"kubernetes.io/projected/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-kube-api-access-b6ntd\") pod \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " Oct 31 19:54:16.580024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.579210 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls\") pod \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " Oct 31 19:54:16.580024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.579484 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-metrics-certs\") pod \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " Oct 31 19:54:16.580024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.579517 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-openshift-service-ca\") pod \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\" (UID: \"7ff3d4c5-512f-4be0-b799-2d2d1527bdee\") " Oct 31 19:54:16.581661 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.581327 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-openshift-service-ca" pod "7ff3d4c5-512f-4be0-b799-2d2d1527bdee" (UID: "7ff3d4c5-512f-4be0-b799-2d2d1527bdee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:16.582890 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.582291 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:54:16.590899 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.590844 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls" pod "7ff3d4c5-512f-4be0-b799-2d2d1527bdee" (UID: "7ff3d4c5-512f-4be0-b799-2d2d1527bdee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:54:16.609256 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.609213 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-kube-api-access-b6ntd" pod "7ff3d4c5-512f-4be0-b799-2d2d1527bdee" (UID: "7ff3d4c5-512f-4be0-b799-2d2d1527bdee"). InnerVolumeSpecName "kube-api-access-b6ntd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:16.609583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.609561 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-metrics-certs" pod "7ff3d4c5-512f-4be0-b799-2d2d1527bdee" (UID: "7ff3d4c5-512f-4be0-b799-2d2d1527bdee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:54:16.689662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.688585 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:16.689662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.688627 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:16.689662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.688642 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6ntd\" (UniqueName: \"kubernetes.io/projected/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-kube-api-access-b6ntd\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:16.689662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.688656 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3d4c5-512f-4be0-b799-2d2d1527bdee-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:16.702346 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.702323 6481 scope.go:122] "RemoveContainer" containerID="e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400" Oct 31 19:54:16.702821 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:16.702801 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400\": container with ID starting with e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400 not found: ID does not exist" containerID="e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400" Oct 31 19:54:16.702967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.702945 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400"} err="failed to get container status \"e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400\": rpc error: code = NotFound desc = could not find container \"e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400\": container with ID starting with e19265ec3b8a5f557118f770f6b00665f68e51fa65c7c50d47c3baf8786d1400 not found: ID does not exist" Oct 31 19:54:16.880779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.880748 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" path="/var/lib/kubelet/pods/2bb6cda9-dd33-4a69-a1b2-8341a79026af/volumes" Oct 31 19:54:16.979105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.979070 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc"] Oct 31 19:54:16.983624 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:16.983601 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84fd8e82-54cd58f9f4-c8xpc"] Oct 31 19:54:18.880424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:18.880386 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff3d4c5-512f-4be0-b799-2d2d1527bdee" path="/var/lib/kubelet/pods/7ff3d4c5-512f-4be0-b799-2d2d1527bdee/volumes" Oct 31 19:54:19.830159 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:19.830122 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl"] Oct 31 19:54:19.830335 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:19.830313 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" containerID="cri-o://a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070" gracePeriod=30 Oct 31 19:54:21.145580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.145546 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw"] Oct 31 19:54:21.146224 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146208 6481 cpu_manager.go:414] "containerMap: removing container" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" Oct 31 19:54:21.146224 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146225 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" Oct 31 19:54:21.146327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146245 6481 cpu_manager.go:414] "containerMap: removing container" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" Oct 31 19:54:21.146327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146252 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" Oct 31 19:54:21.146327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146264 6481 cpu_manager.go:414] "containerMap: removing container" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" Oct 31 19:54:21.146327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146269 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" Oct 31 19:54:21.146327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146280 6481 cpu_manager.go:414] "containerMap: removing container" podUID="7ff3d4c5-512f-4be0-b799-2d2d1527bdee" containerName="maas-api" Oct 31 19:54:21.146327 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146284 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff3d4c5-512f-4be0-b799-2d2d1527bdee" containerName="maas-api" Oct 31 19:54:21.146505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146374 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9733ccf-04a4-4b33-913e-143456ef98a0" containerName="istio-proxy" Oct 31 19:54:21.146505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146385 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b383e843-8685-4714-b21a-ded1019edf5b" containerName="istio-proxy" Oct 31 19:54:21.146505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146393 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb6cda9-dd33-4a69-a1b2-8341a79026af" containerName="istio-proxy" Oct 31 19:54:21.146505 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.146400 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff3d4c5-512f-4be0-b799-2d2d1527bdee" containerName="maas-api" Oct 31 19:54:21.151295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.151277 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.218014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.213107 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-1de410-openshift-default-dockercfg-s9dbm\"" Oct 31 19:54:21.311734 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.311680 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istiod-ca-cert\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.311919 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.311764 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-envoy\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.311919 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.311845 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk97\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-kube-api-access-rqk97\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.312144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.312123 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-token\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.312240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.312165 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-podinfo\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.312240 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.312200 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-credential-socket\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.312339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.312242 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-certs\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.312339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.312277 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-socket\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.312339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.312324 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-data\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.374135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.374088 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw"] Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510298 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-token\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510360 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-podinfo\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510397 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-credential-socket\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510432 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-certs\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510467 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-socket\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510516 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-data\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510669 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istiod-ca-cert\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510717 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-envoy\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.510793 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk97\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-kube-api-access-rqk97\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.512435 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-socket\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.512637 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-credential-socket\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.512814 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-certs\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.513475 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istiod-ca-cert\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.513961 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.513797 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-data\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.559016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.555248 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-podinfo\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.559016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.557795 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-envoy\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.569642 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.569608 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-token\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.575130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.574609 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk97\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-kube-api-access-rqk97\") pod \"e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:21.765989 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:21.765911 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:22.189180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.189154 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw"] Oct 31 19:54:22.191232 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:22.191202 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1bd6b3_6ecf_426d_9b54_7a9fc4135520.slice/crio-568f34da314b81bb1f06246c96ff7a5eb52a1e82cec6b8a0fc307649ea3c1aa3 WatchSource:0}: Error finding container 568f34da314b81bb1f06246c96ff7a5eb52a1e82cec6b8a0fc307649ea3c1aa3: Status 404 returned error can't find the container with id 568f34da314b81bb1f06246c96ff7a5eb52a1e82cec6b8a0fc307649ea3c1aa3 Oct 31 19:54:22.194092 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.193983 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:54:22.194177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.194133 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:54:22.194177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.194165 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:54:22.608092 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.608050 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" event={"ID":"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520","Type":"ContainerStarted","Data":"87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32"} Oct 31 19:54:22.608304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.608108 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" event={"ID":"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520","Type":"ContainerStarted","Data":"568f34da314b81bb1f06246c96ff7a5eb52a1e82cec6b8a0fc307649ea3c1aa3"} Oct 31 19:54:22.766138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.766110 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:22.811026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:22.810956 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podStartSLOduration=1.810937675 podStartE2EDuration="1.810937675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:22.762885815 +0000 UTC m=+2234.003108947" watchObservedRunningTime="2026-10-31 19:54:22.810937675 +0000 UTC m=+2234.051160805" Oct 31 19:54:23.429021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.425616 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j"] Oct 31 19:54:23.741851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.734231 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j"] Oct 31 19:54:23.842323 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.842284 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:23.872340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.872305 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:23.879920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.879850 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:23.881711 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.881678 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:54:23.884155 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.884086 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt"] Oct 31 19:54:23.892222 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.892196 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt"] Oct 31 19:54:23.892416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.892311 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:23.895740 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.895721 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-a5ba62\"" Oct 31 19:54:23.895740 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.895732 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-a5ba62-dockercfg-n5ffg\"" Oct 31 19:54:23.950031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.936911 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:23.950031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:23.937043 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkls\" (UniqueName: \"kubernetes.io/projected/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-kube-api-access-blkls\") pod \"payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:24.039158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.039075 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwfk\" (UniqueName: \"kubernetes.io/projected/56c22de4-a8ca-42ea-991b-857382cba34a-kube-api-access-zvwfk\") pod \"payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.039158 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.039145 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:24.039401 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.039274 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkls\" (UniqueName: \"kubernetes.io/projected/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-kube-api-access-blkls\") pod \"payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:24.040213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.039521 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c22de4-a8ca-42ea-991b-857382cba34a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.040213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.039983 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:24.092263 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.092206 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkls\" (UniqueName: \"kubernetes.io/projected/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-kube-api-access-blkls\") pod \"payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:24.140581 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.140537 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwfk\" (UniqueName: \"kubernetes.io/projected/56c22de4-a8ca-42ea-991b-857382cba34a-kube-api-access-zvwfk\") pod \"payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.140788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.140723 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c22de4-a8ca-42ea-991b-857382cba34a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.141415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.141395 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c22de4-a8ca-42ea-991b-857382cba34a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.197014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.193479 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwfk\" (UniqueName: \"kubernetes.io/projected/56c22de4-a8ca-42ea-991b-857382cba34a-kube-api-access-zvwfk\") pod \"payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.197014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.193734 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:24.210336 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.210306 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.574333 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:24.574304 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b05dd8_cfb2_4dc5_9eb1_17a09058d4bf.slice/crio-6fb13c2926d63c66c485bf64bc038d64cc86be0779a1ad44da9f3173a1acd8a4 WatchSource:0}: Error finding container 6fb13c2926d63c66c485bf64bc038d64cc86be0779a1ad44da9f3173a1acd8a4: Status 404 returned error can't find the container with id 6fb13c2926d63c66c485bf64bc038d64cc86be0779a1ad44da9f3173a1acd8a4 Oct 31 19:54:24.574984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.574833 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j"] Oct 31 19:54:24.621184 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.621159 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt"] Oct 31 19:54:24.622167 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:24.622128 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c22de4_a8ca_42ea_991b_857382cba34a.slice/crio-4ea8627aed1377ce90ee7e5e9ea7264fa9d390a02c1399fd5ca0446dd51d3fef WatchSource:0}: Error finding container 4ea8627aed1377ce90ee7e5e9ea7264fa9d390a02c1399fd5ca0446dd51d3fef: Status 404 returned error can't find the container with id 4ea8627aed1377ce90ee7e5e9ea7264fa9d390a02c1399fd5ca0446dd51d3fef Oct 31 19:54:24.887532 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.887499 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" event={"ID":"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf","Type":"ContainerStarted","Data":"9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c"} Oct 31 19:54:24.887772 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.887550 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" event={"ID":"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf","Type":"ContainerStarted","Data":"6fb13c2926d63c66c485bf64bc038d64cc86be0779a1ad44da9f3173a1acd8a4"} Oct 31 19:54:24.887772 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.887592 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:24.889271 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.889247 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" event={"ID":"56c22de4-a8ca-42ea-991b-857382cba34a","Type":"ContainerStarted","Data":"98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6"} Oct 31 19:54:24.889394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.889284 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" event={"ID":"56c22de4-a8ca-42ea-991b-857382cba34a","Type":"ContainerStarted","Data":"4ea8627aed1377ce90ee7e5e9ea7264fa9d390a02c1399fd5ca0446dd51d3fef"} Oct 31 19:54:24.889440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.889410 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:24.995397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:24.995348 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" podStartSLOduration=1.995332195 podStartE2EDuration="1.995332195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:24.992696301 +0000 UTC m=+2236.232919445" watchObservedRunningTime="2026-10-31 19:54:24.995332195 +0000 UTC m=+2236.235555325" Oct 31 19:54:25.092766 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.092699 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:54:25.099789 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.099737 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" podStartSLOduration=2.099721276 podStartE2EDuration="2.099721276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:25.097672729 +0000 UTC m=+2236.337895860" watchObservedRunningTime="2026-10-31 19:54:25.099721276 +0000 UTC m=+2236.339944404" Oct 31 19:54:25.271194 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.271148 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-credential-socket\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.271870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.271832 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-certs\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.272157 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.272134 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-token\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.272424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.272404 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-podinfo\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.272635 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.272603 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istiod-ca-cert\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.272879 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.272851 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-envoy\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.273069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.273041 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-socket\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.273280 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.273260 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-data\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.273528 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.273501 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-kube-api-access-j89xz\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-kube-api-access-j89xz\") pod \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\" (UID: \"c1d23075-4d02-4dba-83fb-6a7e56a0f31c\") " Oct 31 19:54:25.274367 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.271727 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-credential-socket" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:25.274526 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.274340 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-certs" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:25.275214 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.275142 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-socket" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:25.275596 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.275560 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istiod-ca-cert" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:25.277106 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.276353 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-data" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:25.287458 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.287424 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-envoy" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:25.288562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.288532 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-kube-api-access-j89xz" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "kube-api-access-j89xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:25.302178 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.301052 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-podinfo" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:25.302178 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.301357 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-token" pod "c1d23075-4d02-4dba-83fb-6a7e56a0f31c" (UID: "c1d23075-4d02-4dba-83fb-6a7e56a0f31c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376050 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376083 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376093 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376103 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j89xz\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-kube-api-access-j89xz\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376113 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376122 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376131 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376139 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.376511 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.376148 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d23075-4d02-4dba-83fb-6a7e56a0f31c-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:25.899529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.899498 6481 generic.go:356] "Generic (PLEG): container finished" podID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerID="a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070" exitCode=0 Oct 31 19:54:25.900029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.899569 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" Oct 31 19:54:25.900029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.899585 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" event={"ID":"c1d23075-4d02-4dba-83fb-6a7e56a0f31c","Type":"ContainerDied","Data":"a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070"} Oct 31 19:54:25.900029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.899644 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl" event={"ID":"c1d23075-4d02-4dba-83fb-6a7e56a0f31c","Type":"ContainerDied","Data":"14c2bf5b43c688296b24d0e16bc8130c6eb2f9115751ce66cf667e8fad4f5c50"} Oct 31 19:54:25.900029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.899671 6481 scope.go:122] "RemoveContainer" containerID="a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070" Oct 31 19:54:25.911066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.911040 6481 scope.go:122] "RemoveContainer" containerID="d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3" Oct 31 19:54:25.920391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.920368 6481 scope.go:122] "RemoveContainer" containerID="a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070" Oct 31 19:54:25.920635 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:25.920613 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070\": container with ID starting with a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070 not found: ID does not exist" containerID="a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070" Oct 31 19:54:25.920688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.920643 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070"} err="failed to get container status \"a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070\": rpc error: code = NotFound desc = could not find container \"a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070\": container with ID starting with a14a71cb5a73e7e5eff2e3083fe0ebee1d783fba250c2947f4d375a1132f6070 not found: ID does not exist" Oct 31 19:54:25.920688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.920665 6481 scope.go:122] "RemoveContainer" containerID="d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3" Oct 31 19:54:25.920958 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:25.920940 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3\": container with ID starting with d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3 not found: ID does not exist" containerID="d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3" Oct 31 19:54:25.921032 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:25.920964 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3"} err="failed to get container status \"d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3\": rpc error: code = NotFound desc = could not find container \"d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3\": container with ID starting with d49faac0a1b7789022f14122e8b1db839883cee2e58ae8423e6745e17bbd94f3 not found: ID does not exist" Oct 31 19:54:26.104380 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:26.104341 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl"] Oct 31 19:54:26.150180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:26.150107 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-84fd8e82-openshift-default-59f54fb787-gxktl"] Oct 31 19:54:26.880867 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:26.880833 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" path="/var/lib/kubelet/pods/c1d23075-4d02-4dba-83fb-6a7e56a0f31c/volumes" Oct 31 19:54:27.327007 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:27.326953 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:54:30.692844 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.692801 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s"] Oct 31 19:54:30.694449 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.694418 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" Oct 31 19:54:30.694613 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.694597 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" Oct 31 19:54:30.694737 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.694725 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" Oct 31 19:54:30.694817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.694806 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" Oct 31 19:54:30.695127 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.695109 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" Oct 31 19:54:30.700415 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.700396 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d23075-4d02-4dba-83fb-6a7e56a0f31c" containerName="istio-proxy" Oct 31 19:54:30.734649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.722187 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s"] Oct 31 19:54:30.734649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.722307 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:30.776977 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.776942 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg"] Oct 31 19:54:30.834014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.817629 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-1de410-dockercfg-trns9\"" Oct 31 19:54:30.834014 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.817742 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-1de410\"" Oct 31 19:54:30.873608 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.872213 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:30.890055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.882922 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjr59\" (UniqueName: \"kubernetes.io/projected/ca110c79-031f-4725-8471-6dc641713674-kube-api-access-xjr59\") pod \"payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:30.890055 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.882969 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca110c79-031f-4725-8471-6dc641713674-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:30.913071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.911177 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg"] Oct 31 19:54:30.987628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.987591 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjr59\" (UniqueName: \"kubernetes.io/projected/ca110c79-031f-4725-8471-6dc641713674-kube-api-access-xjr59\") pod \"payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:30.987809 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.987649 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca110c79-031f-4725-8471-6dc641713674-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:30.987906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.987812 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhpl\" (UniqueName: \"kubernetes.io/projected/7532c752-6a84-43a7-ba46-b6c38c03f8c7-kube-api-access-qhhpl\") pod \"payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:30.987968 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.987903 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7532c752-6a84-43a7-ba46-b6c38c03f8c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:30.997088 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:30.996110 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca110c79-031f-4725-8471-6dc641713674-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:31.043636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.043600 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjr59\" (UniqueName: \"kubernetes.io/projected/ca110c79-031f-4725-8471-6dc641713674-kube-api-access-xjr59\") pod \"payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:31.089650 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.089615 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhpl\" (UniqueName: \"kubernetes.io/projected/7532c752-6a84-43a7-ba46-b6c38c03f8c7-kube-api-access-qhhpl\") pod \"payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:31.089810 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.089683 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7532c752-6a84-43a7-ba46-b6c38c03f8c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:31.090435 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.090414 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7532c752-6a84-43a7-ba46-b6c38c03f8c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:31.142654 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.142626 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhpl\" (UniqueName: \"kubernetes.io/projected/7532c752-6a84-43a7-ba46-b6c38c03f8c7-kube-api-access-qhhpl\") pod \"payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:31.175598 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.175562 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:31.265947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.265912 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:31.543077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.543053 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s"] Oct 31 19:54:31.544332 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:31.544300 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca110c79_031f_4725_8471_6dc641713674.slice/crio-cd3dcf8fbcce2cc8187fd4097bd5b366476e14c73712dd32058d9a539e8b0dec WatchSource:0}: Error finding container cd3dcf8fbcce2cc8187fd4097bd5b366476e14c73712dd32058d9a539e8b0dec: Status 404 returned error can't find the container with id cd3dcf8fbcce2cc8187fd4097bd5b366476e14c73712dd32058d9a539e8b0dec Oct 31 19:54:31.652506 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.652078 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg"] Oct 31 19:54:31.653885 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:31.653851 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7532c752_6a84_43a7_ba46_b6c38c03f8c7.slice/crio-51a21341d89a70fa0aae5cf83550f3776e210cdca8b6bf9a1efe69d4cedff2df WatchSource:0}: Error finding container 51a21341d89a70fa0aae5cf83550f3776e210cdca8b6bf9a1efe69d4cedff2df: Status 404 returned error can't find the container with id 51a21341d89a70fa0aae5cf83550f3776e210cdca8b6bf9a1efe69d4cedff2df Oct 31 19:54:31.986725 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.986674 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" event={"ID":"7532c752-6a84-43a7-ba46-b6c38c03f8c7","Type":"ContainerStarted","Data":"2a881f68dbd789f2082daaa642a223edf72bb73a7e1e05a8174d52857a74ff19"} Oct 31 19:54:31.987228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.986745 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" event={"ID":"7532c752-6a84-43a7-ba46-b6c38c03f8c7","Type":"ContainerStarted","Data":"51a21341d89a70fa0aae5cf83550f3776e210cdca8b6bf9a1efe69d4cedff2df"} Oct 31 19:54:31.987228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.986773 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:31.988462 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.988435 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" event={"ID":"ca110c79-031f-4725-8471-6dc641713674","Type":"ContainerStarted","Data":"66a8a1d7257b3a91783518c936b6dcd1fa2918475a74038df16545a114f72fff"} Oct 31 19:54:31.988583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.988485 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" event={"ID":"ca110c79-031f-4725-8471-6dc641713674","Type":"ContainerStarted","Data":"cd3dcf8fbcce2cc8187fd4097bd5b366476e14c73712dd32058d9a539e8b0dec"} Oct 31 19:54:31.988583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:31.988517 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:32.097096 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:32.097051 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" podStartSLOduration=2.097037514 podStartE2EDuration="2.097037514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:32.092890196 +0000 UTC m=+2243.333113324" watchObservedRunningTime="2026-10-31 19:54:32.097037514 +0000 UTC m=+2243.337260642" Oct 31 19:54:32.202488 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:32.202443 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" podStartSLOduration=2.202430391 podStartE2EDuration="2.202430391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:32.198745373 +0000 UTC m=+2243.438968503" watchObservedRunningTime="2026-10-31 19:54:32.202430391 +0000 UTC m=+2243.442653577" Oct 31 19:54:35.902828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:35.902793 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 19:54:35.903338 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:35.902856 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 19:54:36.453341 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.453309 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d"] Oct 31 19:54:36.453545 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.453523 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" containerID="cri-o://a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388" gracePeriod=30 Oct 31 19:54:36.499269 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.499234 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr"] Oct 31 19:54:36.499489 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.499463 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" containerID="cri-o://696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb" gracePeriod=30 Oct 31 19:54:36.502425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.502394 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt"] Oct 31 19:54:36.502649 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.502624 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" containerID="cri-o://21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f" gracePeriod=30 Oct 31 19:54:36.505209 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.505187 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c"] Oct 31 19:54:36.505371 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:36.505353 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" containerID="cri-o://70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97" gracePeriod=30 Oct 31 19:54:39.488956 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.488919 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5"] Oct 31 19:54:39.499704 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.494593 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.552098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.552068 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-7c0d34-serving-cert\"" Oct 31 19:54:39.601670 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.601641 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5"] Oct 31 19:54:39.615212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.615184 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d113153-5532-437d-b16b-c30fa0c8721f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.615390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.615225 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.615456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.615431 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nf2l\" (UniqueName: \"kubernetes.io/projected/6d113153-5532-437d-b16b-c30fa0c8721f-kube-api-access-6nf2l\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.615636 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.615611 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.717090 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.717059 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d113153-5532-437d-b16b-c30fa0c8721f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.717090 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.717093 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.717293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.717197 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nf2l\" (UniqueName: \"kubernetes.io/projected/6d113153-5532-437d-b16b-c30fa0c8721f-kube-api-access-6nf2l\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.717345 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.717327 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.717477 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:39.717457 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-serving-cert: secret "maas-api-e2e-shared-a-w4-7c0d34-serving-cert" not found Oct 31 19:54:39.717554 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:39.717528 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls podName:6d113153-5532-437d-b16b-c30fa0c8721f nodeName:}" failed. No retries permitted until 2026-10-31 19:54:40.21750843 +0000 UTC m=+2251.457731546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls") pod "maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" (UID: "6d113153-5532-437d-b16b-c30fa0c8721f") : secret "maas-api-e2e-shared-a-w4-7c0d34-serving-cert" not found Oct 31 19:54:39.717700 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.717678 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d113153-5532-437d-b16b-c30fa0c8721f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.719473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.719454 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:39.788861 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:39.788708 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nf2l\" (UniqueName: \"kubernetes.io/projected/6d113153-5532-437d-b16b-c30fa0c8721f-kube-api-access-6nf2l\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:40.230696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:40.230652 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:40.233062 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:40.233039 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:40.412303 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:40.412269 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:40.741287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:40.741248 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:54:40.741778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:40.741330 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:54:40.790101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:40.790073 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5"] Oct 31 19:54:40.792065 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:40.792031 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d113153_5532_437d_b16b_c30fa0c8721f.slice/crio-4ebf8611daf0edbd6cd124a44fbf0a729189045baa2f8dcb68ce82cbc66834a6 WatchSource:0}: Error finding container 4ebf8611daf0edbd6cd124a44fbf0a729189045baa2f8dcb68ce82cbc66834a6: Status 404 returned error can't find the container with id 4ebf8611daf0edbd6cd124a44fbf0a729189045baa2f8dcb68ce82cbc66834a6 Oct 31 19:54:41.073652 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.073562 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" event={"ID":"6d113153-5532-437d-b16b-c30fa0c8721f","Type":"ContainerStarted","Data":"4ebf8611daf0edbd6cd124a44fbf0a729189045baa2f8dcb68ce82cbc66834a6"} Oct 31 19:54:41.602150 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.602123 6481 patch_prober.go:29] interesting pod/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 19:54:41.602259 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.602175 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 19:54:41.847177 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.847151 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:54:41.961884 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.961850 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-credential-socket\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.961958 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d382e91b-3398-4c58-9120-64b2e0f32ccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d382e91b-3398-4c58-9120-64b2e0f32ccf-istiod-ca-cert\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962015 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-kube-api-access-p994t\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-kube-api-access-p994t\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962116 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-certs\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962177 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-data\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962340 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962220 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-envoy\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962604 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962346 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-token\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-token\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962604 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962381 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-podinfo\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962604 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962479 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-socket\") pod \"d382e91b-3398-4c58-9120-64b2e0f32ccf\" (UID: \"d382e91b-3398-4c58-9120-64b2e0f32ccf\") " Oct 31 19:54:41.962604 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962504 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d382e91b-3398-4c58-9120-64b2e0f32ccf-istiod-ca-cert" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:41.962795 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.962776 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-credential-socket" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:41.963675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.963239 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:41.963675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.963266 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d382e91b-3398-4c58-9120-64b2e0f32ccf-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:41.963675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.963336 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-certs" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:41.963675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.963349 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-socket" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:41.963675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.963559 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-data" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:41.965413 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.965385 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-podinfo" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:41.965508 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.965436 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-envoy" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:41.965752 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.965729 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-token" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:41.966164 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.966149 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-kube-api-access-p994t" pod "d382e91b-3398-4c58-9120-64b2e0f32ccf" (UID: "d382e91b-3398-4c58-9120-64b2e0f32ccf"). InnerVolumeSpecName "kube-api-access-p994t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:41.998096 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:41.998073 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:54:42.000826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.000798 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:54:42.004066 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.004049 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:54:42.064538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064506 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-socket\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.064696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064568 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-credential-socket\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.064696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064599 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-token\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-token\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.064696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064625 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istiod-ca-cert\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.064696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064657 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-data\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.064931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064755 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-envoy\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.064931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064793 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-podinfo\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.064931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064820 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-podinfo\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.064931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064820 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-socket" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.064931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064851 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-envoy\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.064931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064920 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-envoy\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064926 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-credential-socket" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064940 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-data" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.064972 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-credential-socket\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065053 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istiod-ca-cert" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065093 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-certs\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065135 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-token\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-token\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065189 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-socket\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065217 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-podinfo\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065244 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-certs\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.065286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065279 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-credential-socket\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065328 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-data\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065375 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-kube-api-access-64cfm\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-kube-api-access-64cfm\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065410 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-certs\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065437 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istiod-ca-cert\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065504 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-kube-api-access-f66z7\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-kube-api-access-f66z7\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065544 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-socket\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065652 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-kube-api-access-hd2cc\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-kube-api-access-hd2cc\") pod \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\" (UID: \"7e9eb013-bc86-47e6-9ae7-f14e49981a6f\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065695 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istiod-ca-cert\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065707 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-certs" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065724 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-data\") pod \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\" (UID: \"d0eb8290-aa75-4bc5-ba36-4c090b54cc82\") " Oct 31 19:54:42.065796 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065793 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-token\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-token\") pod \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\" (UID: \"67d20cc7-1819-4c64-8c38-c4ad81f1b469\") " Oct 31 19:54:42.066397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065810 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-credential-socket" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.066397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.065926 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-socket" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.066397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066102 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-socket" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.066397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066300 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-certs" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.066397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066339 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istiod-ca-cert" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:42.066397 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066388 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-certs" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.066720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066531 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-credential-socket" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.066720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066709 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066731 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066747 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066761 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p994t\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-kube-api-access-p994t\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066776 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066790 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066804 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.066829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066820 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066834 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066848 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066862 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066877 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066891 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066905 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066921 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066934 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d382e91b-3398-4c58-9120-64b2e0f32ccf-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066949 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d382e91b-3398-4c58-9120-64b2e0f32ccf-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066963 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.066977 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.067279 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-data" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.067568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.067374 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istiod-ca-cert" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:54:42.068409 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.067638 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-data" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.069290 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.069250 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-podinfo" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:42.069538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.069518 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-envoy" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.069921 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.069897 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-token" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:42.070054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.069899 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-envoy" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.070054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.069901 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-podinfo" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:42.070370 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.070332 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-kube-api-access-f66z7" pod "67d20cc7-1819-4c64-8c38-c4ad81f1b469" (UID: "67d20cc7-1819-4c64-8c38-c4ad81f1b469"). InnerVolumeSpecName "kube-api-access-f66z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:42.070370 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.070363 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-kube-api-access-64cfm" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "kube-api-access-64cfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:42.070522 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.070420 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-token" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:42.070580 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.070542 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-token" pod "d0eb8290-aa75-4bc5-ba36-4c090b54cc82" (UID: "d0eb8290-aa75-4bc5-ba36-4c090b54cc82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:42.070822 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.070803 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-envoy" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:54:42.071067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.071046 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-podinfo" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:54:42.071531 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.071513 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-kube-api-access-hd2cc" pod "7e9eb013-bc86-47e6-9ae7-f14e49981a6f" (UID: "7e9eb013-bc86-47e6-9ae7-f14e49981a6f"). InnerVolumeSpecName "kube-api-access-hd2cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:54:42.083338 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.083316 6481 generic.go:356] "Generic (PLEG): container finished" podID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerID="a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388" exitCode=0 Oct 31 19:54:42.083425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.083370 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" Oct 31 19:54:42.083425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.083404 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" event={"ID":"67d20cc7-1819-4c64-8c38-c4ad81f1b469","Type":"ContainerDied","Data":"a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388"} Oct 31 19:54:42.083534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.083457 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d" event={"ID":"67d20cc7-1819-4c64-8c38-c4ad81f1b469","Type":"ContainerDied","Data":"a14b29f762839e94e88e30b7555b80a3e65dd7ba10fa6caf17baa067d9b6befb"} Oct 31 19:54:42.083534 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.083487 6481 scope.go:122] "RemoveContainer" containerID="a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388" Oct 31 19:54:42.084936 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.084908 6481 generic.go:356] "Generic (PLEG): container finished" podID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerID="696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb" exitCode=0 Oct 31 19:54:42.085049 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.084962 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" Oct 31 19:54:42.085049 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.084989 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" event={"ID":"d382e91b-3398-4c58-9120-64b2e0f32ccf","Type":"ContainerDied","Data":"696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb"} Oct 31 19:54:42.085175 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.085079 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr" event={"ID":"d382e91b-3398-4c58-9120-64b2e0f32ccf","Type":"ContainerDied","Data":"0f7d207e6ead784eac05b502523cdd09664430a795d98aed8e0e641078eedb1f"} Oct 31 19:54:42.086758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.086736 6481 generic.go:356] "Generic (PLEG): container finished" podID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerID="70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97" exitCode=0 Oct 31 19:54:42.086848 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.086795 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" Oct 31 19:54:42.086848 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.086814 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" event={"ID":"7e9eb013-bc86-47e6-9ae7-f14e49981a6f","Type":"ContainerDied","Data":"70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97"} Oct 31 19:54:42.086985 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.086862 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c" event={"ID":"7e9eb013-bc86-47e6-9ae7-f14e49981a6f","Type":"ContainerDied","Data":"3fadfa66e64a15187f2a3f964c9a537668c0738ed590481858f4fba342049ac4"} Oct 31 19:54:42.088486 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.088461 6481 generic.go:356] "Generic (PLEG): container finished" podID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerID="21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f" exitCode=0 Oct 31 19:54:42.088587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.088542 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" event={"ID":"d0eb8290-aa75-4bc5-ba36-4c090b54cc82","Type":"ContainerDied","Data":"21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f"} Oct 31 19:54:42.088587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.088580 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" event={"ID":"d0eb8290-aa75-4bc5-ba36-4c090b54cc82","Type":"ContainerDied","Data":"956fd475488c24fac67843a11bba3d2f5fd998a523bbcb7e861a0f29d73b61f7"} Oct 31 19:54:42.088696 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.088546 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt" Oct 31 19:54:42.099551 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.097664 6481 scope.go:122] "RemoveContainer" containerID="5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e" Oct 31 19:54:42.099551 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.098464 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" event={"ID":"6d113153-5532-437d-b16b-c30fa0c8721f","Type":"ContainerStarted","Data":"95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5"} Oct 31 19:54:42.099551 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.099222 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:42.110705 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.110678 6481 scope.go:122] "RemoveContainer" containerID="a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388" Oct 31 19:54:42.111051 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:42.111020 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388\": container with ID starting with a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388 not found: ID does not exist" containerID="a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388" Oct 31 19:54:42.111144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.111062 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388"} err="failed to get container status \"a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388\": rpc error: code = NotFound desc = could not find container \"a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388\": container with ID starting with a73a256e0a1aca1dc1c416cb56677238afe86d71a2b39dddc7385b87e22f6388 not found: ID does not exist" Oct 31 19:54:42.111144 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.111086 6481 scope.go:122] "RemoveContainer" containerID="5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e" Oct 31 19:54:42.118772 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:42.118741 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e\": container with ID starting with 5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e not found: ID does not exist" containerID="5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e" Oct 31 19:54:42.119326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.118785 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e"} err="failed to get container status \"5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e\": rpc error: code = NotFound desc = could not find container \"5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e\": container with ID starting with 5281c5d9e08f9173bed736d8115b9cd2fffd6ab9478aacaa55ac6f5e7302fe3e not found: ID does not exist" Oct 31 19:54:42.119326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.118966 6481 scope.go:122] "RemoveContainer" containerID="696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb" Oct 31 19:54:42.144215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.144189 6481 scope.go:122] "RemoveContainer" containerID="696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb" Oct 31 19:54:42.144794 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:42.144768 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb\": container with ID starting with 696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb not found: ID does not exist" containerID="696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb" Oct 31 19:54:42.144909 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.144806 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb"} err="failed to get container status \"696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb\": rpc error: code = NotFound desc = could not find container \"696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb\": container with ID starting with 696e27efde64145c2eaa9d3094e6d1967f06615c7ced4de4c22df993dcd3c3eb not found: ID does not exist" Oct 31 19:54:42.144909 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.144830 6481 scope.go:122] "RemoveContainer" containerID="70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97" Oct 31 19:54:42.156940 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.156922 6481 scope.go:122] "RemoveContainer" containerID="70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97" Oct 31 19:54:42.162130 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:42.162105 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97\": container with ID starting with 70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97 not found: ID does not exist" containerID="70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97" Oct 31 19:54:42.162248 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.162135 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97"} err="failed to get container status \"70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97\": rpc error: code = NotFound desc = could not find container \"70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97\": container with ID starting with 70de082ab0c00c3a47894caad226046ba7b5a4bf040f18e13a80610b05f5ed97 not found: ID does not exist" Oct 31 19:54:42.162248 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.162153 6481 scope.go:122] "RemoveContainer" containerID="21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f" Oct 31 19:54:42.168797 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.168772 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.168947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.168921 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.168950 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64cfm\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-kube-api-access-64cfm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.168966 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66z7\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-kube-api-access-f66z7\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.168980 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd2cc\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-kube-api-access-hd2cc\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169011 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169025 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169040 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169051 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169059 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9eb013-bc86-47e6-9ae7-f14e49981a6f-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169068 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169076 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169085 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d20cc7-1819-4c64-8c38-c4ad81f1b469-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169093 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.169098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.169101 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0eb8290-aa75-4bc5-ba36-4c090b54cc82-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:54:42.176357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.176335 6481 scope.go:122] "RemoveContainer" containerID="21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f" Oct 31 19:54:42.176658 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:42.176634 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f\": container with ID starting with 21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f not found: ID does not exist" containerID="21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f" Oct 31 19:54:42.176743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.176665 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f"} err="failed to get container status \"21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f\": rpc error: code = NotFound desc = could not find container \"21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f\": container with ID starting with 21bb9debd2696eb4bf7dab241778bb111c344de0973eedc03b505f3b6d25683f not found: ID does not exist" Oct 31 19:54:42.466336 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.466294 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c"] Oct 31 19:54:42.483264 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.483229 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-5w42c"] Oct 31 19:54:42.575268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.575227 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d"] Oct 31 19:54:42.578633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.578607 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-mp25d"] Oct 31 19:54:42.714662 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.714623 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr"] Oct 31 19:54:42.732514 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.730785 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-7fbdr"] Oct 31 19:54:42.804465 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.804433 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt"] Oct 31 19:54:42.807974 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.807950 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a4438-openshift-default-6885fcb8f5-lv6rt"] Oct 31 19:54:42.883786 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.883749 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" path="/var/lib/kubelet/pods/67d20cc7-1819-4c64-8c38-c4ad81f1b469/volumes" Oct 31 19:54:42.884381 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.884363 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" path="/var/lib/kubelet/pods/7e9eb013-bc86-47e6-9ae7-f14e49981a6f/volumes" Oct 31 19:54:42.884864 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.884851 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" path="/var/lib/kubelet/pods/d0eb8290-aa75-4bc5-ba36-4c090b54cc82/volumes" Oct 31 19:54:42.885684 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.885668 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" path="/var/lib/kubelet/pods/d382e91b-3398-4c58-9120-64b2e0f32ccf/volumes" Oct 31 19:54:42.913424 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:42.913368 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" podStartSLOduration=3.09856654 podStartE2EDuration="3.913352098s" totalImagesPullingTime="814.785558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:42.912250592 +0000 UTC m=+2254.152473722" watchObservedRunningTime="2026-10-31 19:54:42.913352098 +0000 UTC m=+2254.153575232" Oct 31 19:54:43.009213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:43.009134 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 19:54:43.009213 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:43.009200 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 19:54:44.520050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520019 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg"] Oct 31 19:54:44.520644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520626 6481 cpu_manager.go:414] "containerMap: removing container" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" Oct 31 19:54:44.520644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520642 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" Oct 31 19:54:44.520807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520663 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" Oct 31 19:54:44.520807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520670 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" Oct 31 19:54:44.520807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520676 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" Oct 31 19:54:44.520807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520681 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" Oct 31 19:54:44.520807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520688 6481 cpu_manager.go:414] "containerMap: removing container" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" Oct 31 19:54:44.520807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520693 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" Oct 31 19:54:44.520807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520805 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" Oct 31 19:54:44.521039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520814 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9eb013-bc86-47e6-9ae7-f14e49981a6f" containerName="istio-proxy" Oct 31 19:54:44.521039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520824 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0eb8290-aa75-4bc5-ba36-4c090b54cc82" containerName="istio-proxy" Oct 31 19:54:44.521039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.520830 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d382e91b-3398-4c58-9120-64b2e0f32ccf" containerName="istio-proxy" Oct 31 19:54:44.526161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.526143 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.577586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.577563 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9902ffd9-serving-cert\"" Oct 31 19:54:44.614664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.614627 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-metrics-certs\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.614831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.614759 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-openshift-service-ca\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.614831 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.614792 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.614990 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.614970 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9xz\" (UniqueName: \"kubernetes.io/projected/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-kube-api-access-rh9xz\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.626651 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.626625 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg"] Oct 31 19:54:44.716348 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.716318 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9xz\" (UniqueName: \"kubernetes.io/projected/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-kube-api-access-rh9xz\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.716512 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.716399 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-metrics-certs\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.716612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.716583 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-openshift-service-ca\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.716686 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.716644 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.716747 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:44.716725 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-serving-cert: secret "maas-api-e2e-mt-9902ffd9-serving-cert" not found Oct 31 19:54:44.716809 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:44.716789 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls podName:05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd nodeName:}" failed. No retries permitted until 2026-10-31 19:54:45.216769993 +0000 UTC m=+2256.456993119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls") pod "maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" (UID: "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd") : secret "maas-api-e2e-mt-9902ffd9-serving-cert" not found Oct 31 19:54:44.717297 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.717260 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-openshift-service-ca\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.718967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.718949 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-metrics-certs\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:44.798908 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:44.794502 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9xz\" (UniqueName: \"kubernetes.io/projected/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-kube-api-access-rh9xz\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:45.230394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:45.230359 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:45.232903 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:45.232879 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:45.440791 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:45.440749 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:45.819301 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:45.819271 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg"] Oct 31 19:54:45.820633 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:54:45.820603 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05edfa8f_5c0d_4cf5_8e01_3e36de09a0cd.slice/crio-68f0aaf6671e2c79e39216ba3d85723ced2086cf64d8e9f818626a8bc3683ab9 WatchSource:0}: Error finding container 68f0aaf6671e2c79e39216ba3d85723ced2086cf64d8e9f818626a8bc3683ab9: Status 404 returned error can't find the container with id 68f0aaf6671e2c79e39216ba3d85723ced2086cf64d8e9f818626a8bc3683ab9 Oct 31 19:54:46.151130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:46.151044 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" event={"ID":"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd","Type":"ContainerStarted","Data":"68f0aaf6671e2c79e39216ba3d85723ced2086cf64d8e9f818626a8bc3683ab9"} Oct 31 19:54:47.160425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:47.160323 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" event={"ID":"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd","Type":"ContainerStarted","Data":"b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b"} Oct 31 19:54:47.160425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:47.160394 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:47.277509 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:47.277456 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" podStartSLOduration=2.377394732 podStartE2EDuration="3.277443913s" totalImagesPullingTime="900.049181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:47.276162575 +0000 UTC m=+2258.516385712" watchObservedRunningTime="2026-10-31 19:54:47.277443913 +0000 UTC m=+2258.517667083" Oct 31 19:54:48.122525 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:48.122496 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 19:54:53.172140 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:54:53.172042 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 19:54:57.324857 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:54:57.324817 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:55:07.603457 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.603420 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7"] Oct 31 19:55:07.604131 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.604114 6481 cpu_manager.go:414] "containerMap: removing container" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" Oct 31 19:55:07.604200 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.604134 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" Oct 31 19:55:07.604280 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.604270 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d20cc7-1819-4c64-8c38-c4ad81f1b469" containerName="istio-proxy" Oct 31 19:55:07.608286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.608262 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.642356 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642322 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istiod-ca-cert\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.642539 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642512 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-token\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.642640 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642625 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-data\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.642687 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642672 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-socket\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.642742 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642707 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-podinfo\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.642852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642830 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvfwq\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-kube-api-access-gvfwq\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.642941 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642926 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-envoy\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.643017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.642977 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-certs\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.643180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.643163 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-credential-socket\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.656778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.656751 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6eeb83-openshift-default-dockercfg-p8h2s\"" Oct 31 19:55:07.706963 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.706929 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7"] Oct 31 19:55:07.857086 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.856933 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-envoy\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857086 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857066 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-certs\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857217 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-credential-socket\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857251 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istiod-ca-cert\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857456 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857377 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-token\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857457 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-data\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857518 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857502 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-socket\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857627 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857529 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-podinfo\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857627 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857569 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvfwq\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-kube-api-access-gvfwq\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857921 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857773 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-credential-socket\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.857921 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.857853 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-certs\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.858130 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.858066 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-data\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.858226 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.858204 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-socket\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.858418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.858398 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istiod-ca-cert\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.862061 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.861480 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-podinfo\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.862369 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.862349 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-envoy\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.912688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.912652 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvfwq\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-kube-api-access-gvfwq\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.912843 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.912744 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-token\") pod \"e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:07.923464 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:07.923352 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:08.408543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:08.408513 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7"] Oct 31 19:55:08.409460 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:55:08.409430 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f41f49d_f53c_4dc9_a9d7_8b242c3c49f5.slice/crio-7501991766a284100cce78b4d472a74aefbda237f4f232a85e0378b6438fca50 WatchSource:0}: Error finding container 7501991766a284100cce78b4d472a74aefbda237f4f232a85e0378b6438fca50: Status 404 returned error can't find the container with id 7501991766a284100cce78b4d472a74aefbda237f4f232a85e0378b6438fca50 Oct 31 19:55:08.412102 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:08.412071 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:08.412180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:08.412139 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:08.412180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:08.412169 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:09.337238 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:09.337197 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" event={"ID":"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5","Type":"ContainerStarted","Data":"243fce928ba1b7c0ddaada721e4139d003fc6d568ff8ac19fe10626a8a0779b7"} Oct 31 19:55:09.337626 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:09.337265 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" event={"ID":"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5","Type":"ContainerStarted","Data":"7501991766a284100cce78b4d472a74aefbda237f4f232a85e0378b6438fca50"} Oct 31 19:55:09.496321 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:09.496274 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podStartSLOduration=2.496260453 podStartE2EDuration="2.496260453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:09.493018016 +0000 UTC m=+2280.733241146" watchObservedRunningTime="2026-10-31 19:55:09.496260453 +0000 UTC m=+2280.736483582" Oct 31 19:55:09.923527 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:09.923498 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:09.929094 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:09.929069 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:10.344121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:10.344090 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:10.345179 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:10.345160 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:55:14.375759 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:14.375725 6481 generic.go:356] "Generic (PLEG): container finished" podID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerID="146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e" exitCode=0 Oct 31 19:55:14.376228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:14.375772 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" event={"ID":"98cabd44-169b-40d1-82a5-eb3588801cbe","Type":"ContainerDied","Data":"146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e"} Oct 31 19:55:14.376228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:14.376180 6481 scope.go:122] "RemoveContainer" containerID="146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e" Oct 31 19:55:14.377684 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:14.377655 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:14.377757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:14.377727 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:14.377793 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:14.377758 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:15.401008 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:15.400973 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" event={"ID":"98cabd44-169b-40d1-82a5-eb3588801cbe","Type":"ContainerStarted","Data":"a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c"} Oct 31 19:55:16.108804 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.108760 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv"] Oct 31 19:55:16.121982 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.121950 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.148111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.148086 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv"] Oct 31 19:55:16.189451 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.189411 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6eeb83\"" Oct 31 19:55:16.222019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.221056 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hc54\" (UniqueName: \"kubernetes.io/projected/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-kube-api-access-9hc54\") pod \"payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.222019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.221204 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.319372 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.313026 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v"] Oct 31 19:55:16.342024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.341425 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hc54\" (UniqueName: \"kubernetes.io/projected/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-kube-api-access-9hc54\") pod \"payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.342024 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.341673 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.385068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.384019 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:55:16.385068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.384096 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 19:55:16.487167 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.474340 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:55:16.495050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.494164 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v"] Oct 31 19:55:16.495050 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.494084 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.504021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.489379 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.504852 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.504502 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hc54\" (UniqueName: \"kubernetes.io/projected/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-kube-api-access-9hc54\") pod \"payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.513059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.505524 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:16.544959 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.544932 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24f1601-6686-49d2-b525-5b302a70c2cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.545101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.545044 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4psj\" (UniqueName: \"kubernetes.io/projected/b24f1601-6686-49d2-b525-5b302a70c2cf-kube-api-access-v4psj\") pod \"payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.545360 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.545342 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6eeb83-dockercfg-sczhp\"" Oct 31 19:55:16.646743 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.646657 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24f1601-6686-49d2-b525-5b302a70c2cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.646906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.646754 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4psj\" (UniqueName: \"kubernetes.io/projected/b24f1601-6686-49d2-b525-5b302a70c2cf-kube-api-access-v4psj\") pod \"payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.647551 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.647523 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24f1601-6686-49d2-b525-5b302a70c2cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.700846 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.700815 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4psj\" (UniqueName: \"kubernetes.io/projected/b24f1601-6686-49d2-b525-5b302a70c2cf-kube-api-access-v4psj\") pod \"payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.836033 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.835238 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:16.914574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:16.914548 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv"] Oct 31 19:55:17.294163 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.294136 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v"] Oct 31 19:55:17.297415 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:55:17.297383 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24f1601_6686_49d2_b525_5b302a70c2cf.slice/crio-ebe5f0147150290bbd521acacc97424ae6b5ae59e0ddd5ae86b6459ba9fca3aa WatchSource:0}: Error finding container ebe5f0147150290bbd521acacc97424ae6b5ae59e0ddd5ae86b6459ba9fca3aa: Status 404 returned error can't find the container with id ebe5f0147150290bbd521acacc97424ae6b5ae59e0ddd5ae86b6459ba9fca3aa Oct 31 19:55:17.351299 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.351276 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:55:17.351414 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.351320 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 19:55:17.527105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.527055 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" event={"ID":"b24f1601-6686-49d2-b525-5b302a70c2cf","Type":"ContainerStarted","Data":"eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12"} Oct 31 19:55:17.527589 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.527125 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:17.527589 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.527151 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" event={"ID":"b24f1601-6686-49d2-b525-5b302a70c2cf","Type":"ContainerStarted","Data":"ebe5f0147150290bbd521acacc97424ae6b5ae59e0ddd5ae86b6459ba9fca3aa"} Oct 31 19:55:17.528799 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.528771 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" event={"ID":"3def61aa-6d85-4e51-88fa-cf4866d4a9f0","Type":"ContainerStarted","Data":"07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47"} Oct 31 19:55:17.528927 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.528818 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" event={"ID":"3def61aa-6d85-4e51-88fa-cf4866d4a9f0","Type":"ContainerStarted","Data":"83f4604f23f12bafb4932e50b7fe578769a1b22ccdc3dd04294e1c874c5cf4cc"} Oct 31 19:55:17.528927 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.528847 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:17.646510 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.646410 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" podStartSLOduration=1.646395764 podStartE2EDuration="1.646395764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:17.641873562 +0000 UTC m=+2288.882096690" watchObservedRunningTime="2026-10-31 19:55:17.646395764 +0000 UTC m=+2288.886618893" Oct 31 19:55:17.748798 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:17.748735 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" podStartSLOduration=1.748716252 podStartE2EDuration="1.748716252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:17.74704154 +0000 UTC m=+2288.987264669" watchObservedRunningTime="2026-10-31 19:55:17.748716252 +0000 UTC m=+2288.988939382" Oct 31 19:55:18.350709 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:18.350665 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:55:18.350917 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:18.350729 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 19:55:19.350470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:19.350441 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:55:19.350875 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:19.350486 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 19:55:20.350757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:20.350722 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:55:20.351207 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:20.350775 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 19:55:21.335416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.335375 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9"] Oct 31 19:55:21.339912 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.339893 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.356447 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.356421 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:55:21.356828 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.356712 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:55:21.357565 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.357545 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 19:55:21.373127 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.373097 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.373263 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.373136 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798bcb87-4cb1-4aac-8e6c-04033dff6888-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.373357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.373337 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.373612 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.373588 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7m4\" (UniqueName: \"kubernetes.io/projected/798bcb87-4cb1-4aac-8e6c-04033dff6888-kube-api-access-vr7m4\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.436149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.436108 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-a5ba62-serving-cert\"" Oct 31 19:55:21.439543 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.439519 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9"] Oct 31 19:55:21.475244 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.475209 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.475440 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.475392 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7m4\" (UniqueName: \"kubernetes.io/projected/798bcb87-4cb1-4aac-8e6c-04033dff6888-kube-api-access-vr7m4\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.475698 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.475655 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.475895 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.475869 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798bcb87-4cb1-4aac-8e6c-04033dff6888-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.475895 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:55:21.475886 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-serving-cert: secret "maas-api-e2e-shared-a-w6-a5ba62-serving-cert" not found Oct 31 19:55:21.476109 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:55:21.475948 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls podName:798bcb87-4cb1-4aac-8e6c-04033dff6888 nodeName:}" failed. No retries permitted until 2026-10-31 19:55:21.975929063 +0000 UTC m=+2293.216152189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls") pod "maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" (UID: "798bcb87-4cb1-4aac-8e6c-04033dff6888") : secret "maas-api-e2e-shared-a-w6-a5ba62-serving-cert" not found Oct 31 19:55:21.476562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.476539 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798bcb87-4cb1-4aac-8e6c-04033dff6888-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.480571 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.480542 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.528819 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.528784 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7m4\" (UniqueName: \"kubernetes.io/projected/798bcb87-4cb1-4aac-8e6c-04033dff6888-kube-api-access-vr7m4\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.986591 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.986550 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:21.988967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:21.988936 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:22.256054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:22.255931 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:22.627813 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:22.627786 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9"] Oct 31 19:55:22.629635 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:55:22.629607 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798bcb87_4cb1_4aac_8e6c_04033dff6888.slice/crio-2438551fa366937b3cb9ad4f8b413db1c785c673bb75ac7ed6fa47acb717997c WatchSource:0}: Error finding container 2438551fa366937b3cb9ad4f8b413db1c785c673bb75ac7ed6fa47acb717997c: Status 404 returned error can't find the container with id 2438551fa366937b3cb9ad4f8b413db1c785c673bb75ac7ed6fa47acb717997c Oct 31 19:55:23.579920 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:23.579885 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" event={"ID":"798bcb87-4cb1-4aac-8e6c-04033dff6888","Type":"ContainerStarted","Data":"2438551fa366937b3cb9ad4f8b413db1c785c673bb75ac7ed6fa47acb717997c"} Oct 31 19:55:24.609019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:24.607447 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" event={"ID":"798bcb87-4cb1-4aac-8e6c-04033dff6888","Type":"ContainerStarted","Data":"5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a"} Oct 31 19:55:24.609019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:24.608730 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:24.766016 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:24.742807 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" podStartSLOduration=2.830753204 podStartE2EDuration="3.742788421s" totalImagesPullingTime="912.035217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:24.73980802 +0000 UTC m=+2295.980031150" watchObservedRunningTime="2026-10-31 19:55:24.742788421 +0000 UTC m=+2295.983011554" Oct 31 19:55:25.961395 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:25.961360 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj"] Oct 31 19:55:25.966120 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:25.966103 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.014610 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.014581 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5370f5-openshift-default-dockercfg-ps8qc\"" Oct 31 19:55:26.059758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.059719 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-envoy\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.059933 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.059780 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-data\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.059933 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.059858 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-certs\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.060062 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.059958 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-token\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.060062 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.059986 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfbp\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-kube-api-access-pnfbp\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.060207 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.060188 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-credential-socket\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.060326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.060311 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-socket\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.060390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.060341 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee90507-77f4-497c-b6b6-70b47a953dec-istiod-ca-cert\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.060390 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.060375 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-podinfo\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.064494 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.064470 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj"] Oct 31 19:55:26.164756 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.164627 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-envoy\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.165619 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.165577 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-data\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.166214 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.166195 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-certs\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.166794 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.166772 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-token\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.167569 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.167545 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfbp\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-kube-api-access-pnfbp\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.168334 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.168265 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-credential-socket\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.169042 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.169022 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-socket\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.169784 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.169763 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee90507-77f4-497c-b6b6-70b47a953dec-istiod-ca-cert\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.173082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.172316 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-podinfo\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.173548 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.169700 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-socket\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.173688 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.166140 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-data\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.173811 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.170661 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee90507-77f4-497c-b6b6-70b47a953dec-istiod-ca-cert\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.173904 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.168821 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-credential-socket\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.173984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.166624 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-certs\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.177228 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.177127 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-podinfo\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.200281 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.200243 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-envoy\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.224166 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.223861 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfbp\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-kube-api-access-pnfbp\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.228395 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.228365 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-token\") pod \"e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.282690 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.282652 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:26.740592 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.740566 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj"] Oct 31 19:55:26.742137 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:55:26.742104 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee90507_77f4_497c_b6b6_70b47a953dec.slice/crio-8dab9ba6c94033a363b2792362383d8d4d93d6db4da3d4077f4fa9b845b614d7 WatchSource:0}: Error finding container 8dab9ba6c94033a363b2792362383d8d4d93d6db4da3d4077f4fa9b845b614d7: Status 404 returned error can't find the container with id 8dab9ba6c94033a363b2792362383d8d4d93d6db4da3d4077f4fa9b845b614d7 Oct 31 19:55:26.745026 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.744977 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:26.745131 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.745063 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:26.745131 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:26.745096 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:55:27.325409 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:55:27.325366 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:55:27.630728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:27.630639 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" event={"ID":"5ee90507-77f4-497c-b6b6-70b47a953dec","Type":"ContainerStarted","Data":"f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678"} Oct 31 19:55:27.630728 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:27.630711 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" event={"ID":"5ee90507-77f4-497c-b6b6-70b47a953dec","Type":"ContainerStarted","Data":"8dab9ba6c94033a363b2792362383d8d4d93d6db4da3d4077f4fa9b845b614d7"} Oct 31 19:55:27.845287 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:27.845230 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podStartSLOduration=2.8452145829999997 podStartE2EDuration="2.845214583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:27.797894129 +0000 UTC m=+2299.038117260" watchObservedRunningTime="2026-10-31 19:55:27.845214583 +0000 UTC m=+2299.085437713" Oct 31 19:55:28.283881 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:28.283840 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:28.289778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:28.289749 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:28.537984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:28.537906 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 19:55:28.537984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:28.537977 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 19:55:28.637068 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:28.636915 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:28.637748 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:28.637727 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:55:31.625538 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:31.625506 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 19:55:36.653723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.653688 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx"] Oct 31 19:55:36.658450 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.658429 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.673661 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.673638 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.673788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.673676 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.673788 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.673761 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10361c-e8f2-4c07-a044-0d8340acb152-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.674115 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.674094 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgjk\" (UniqueName: \"kubernetes.io/projected/fb10361c-e8f2-4c07-a044-0d8340acb152-kube-api-access-cvgjk\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.711919 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.711888 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx"] Oct 31 19:55:36.753644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.753604 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-1de410-serving-cert\"" Oct 31 19:55:36.774944 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.774914 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.775078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.774953 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.775078 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.774981 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10361c-e8f2-4c07-a044-0d8340acb152-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.775203 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:55:36.775089 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-serving-cert: secret "maas-api-e2e-shared-a-w1-1de410-serving-cert" not found Oct 31 19:55:36.775203 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:55:36.775151 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls podName:fb10361c-e8f2-4c07-a044-0d8340acb152 nodeName:}" failed. No retries permitted until 2026-10-31 19:55:37.275130915 +0000 UTC m=+2308.515354042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls") pod "maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" (UID: "fb10361c-e8f2-4c07-a044-0d8340acb152") : secret "maas-api-e2e-shared-a-w1-1de410-serving-cert" not found Oct 31 19:55:36.775319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.775275 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgjk\" (UniqueName: \"kubernetes.io/projected/fb10361c-e8f2-4c07-a044-0d8340acb152-kube-api-access-cvgjk\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.776106 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.776082 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10361c-e8f2-4c07-a044-0d8340acb152-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.777599 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.777577 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:36.837819 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:36.837790 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgjk\" (UniqueName: \"kubernetes.io/projected/fb10361c-e8f2-4c07-a044-0d8340acb152-kube-api-access-cvgjk\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:37.286953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:37.286915 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:37.289448 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:37.289428 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:37.573365 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:37.573271 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:37.945478 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:37.945447 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx"] Oct 31 19:55:37.946813 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:55:37.946780 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb10361c_e8f2_4c07_a044_0d8340acb152.slice/crio-616cbd2a3e51856bd0481c927be30024eb653b38be3f46f2ac77719bc16bd6e5 WatchSource:0}: Error finding container 616cbd2a3e51856bd0481c927be30024eb653b38be3f46f2ac77719bc16bd6e5: Status 404 returned error can't find the container with id 616cbd2a3e51856bd0481c927be30024eb653b38be3f46f2ac77719bc16bd6e5 Oct 31 19:55:38.719286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:38.719243 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" event={"ID":"fb10361c-e8f2-4c07-a044-0d8340acb152","Type":"ContainerStarted","Data":"616cbd2a3e51856bd0481c927be30024eb653b38be3f46f2ac77719bc16bd6e5"} Oct 31 19:55:39.805664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:39.802488 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" event={"ID":"fb10361c-e8f2-4c07-a044-0d8340acb152","Type":"ContainerStarted","Data":"e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7"} Oct 31 19:55:39.805664 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:39.803988 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:39.915029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:39.914959 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" podStartSLOduration=3.099453619 podStartE2EDuration="3.914945098s" totalImagesPullingTime="815.491479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:39.912201187 +0000 UTC m=+2311.152424328" watchObservedRunningTime="2026-10-31 19:55:39.914945098 +0000 UTC m=+2311.155168254" Oct 31 19:55:46.836403 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:46.836367 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 19:55:57.325089 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:55:57.325046 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:55:59.288168 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.288089 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk"] Oct 31 19:55:59.294042 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.293988 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.394018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.382657 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9a8346-openshift-default-dockercfg-srsdb\"" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456464 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-podinfo\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456583 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrsb\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-kube-api-access-fwrsb\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456627 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-token\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456663 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-envoy\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456704 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-data\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456779 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-certs\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456829 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-credential-socket\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.462018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.456866 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istiod-ca-cert\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.513015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.504331 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-socket\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.612437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.612346 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-podinfo\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.613368 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.613252 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrsb\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-kube-api-access-fwrsb\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.613604 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.613584 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-token\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.613818 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.613802 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-envoy\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.613967 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.613948 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-data\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.614184 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.614162 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-certs\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.614396 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.614338 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-credential-socket\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.614549 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.614533 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istiod-ca-cert\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.614927 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.614906 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-socket\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.615695 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.615675 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-socket\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.617180 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.617153 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-data\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.618936 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.618907 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-certs\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.619957 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.619937 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-credential-socket\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.689918 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.623682 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istiod-ca-cert\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.689918 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.660577 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-envoy\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.691293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.678082 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-token\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.691293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.684818 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk"] Oct 31 19:55:59.691293 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.685540 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrsb\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-kube-api-access-fwrsb\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.694737 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.694656 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-podinfo\") pod \"e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:55:59.908161 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:55:59.908075 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:56:00.349225 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:56:00.346113 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ef4a4d_8d1c_4399_8d38_a8cfdea2ce6e.slice/crio-5b4e83fcf367f380f7c9f36fb9405598c92740043455809d90864a30966057a8 WatchSource:0}: Error finding container 5b4e83fcf367f380f7c9f36fb9405598c92740043455809d90864a30966057a8: Status 404 returned error can't find the container with id 5b4e83fcf367f380f7c9f36fb9405598c92740043455809d90864a30966057a8 Oct 31 19:56:00.351171 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:00.351138 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk"] Oct 31 19:56:00.363832 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:00.363758 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:00.364035 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:00.363838 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:00.364035 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:00.363871 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:01.102530 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:01.102495 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" event={"ID":"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e","Type":"ContainerStarted","Data":"0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb"} Oct 31 19:56:01.102726 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:01.102555 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" event={"ID":"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e","Type":"ContainerStarted","Data":"5b4e83fcf367f380f7c9f36fb9405598c92740043455809d90864a30966057a8"} Oct 31 19:56:01.212190 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:01.212137 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podStartSLOduration=2.212123678 podStartE2EDuration="2.212123678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:01.210741605 +0000 UTC m=+2332.450964736" watchObservedRunningTime="2026-10-31 19:56:01.212123678 +0000 UTC m=+2332.452346873" Oct 31 19:56:01.908980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:01.908943 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:56:01.914614 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:01.914588 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:56:02.109188 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:02.109152 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:56:02.110205 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:02.110182 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:56:10.180586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.180551 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf"] Oct 31 19:56:10.183776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.183754 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.281437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.281404 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9902ffd9\"/\"kube-root-ca.crt\"" Oct 31 19:56:10.281437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.281436 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9902ffd9\"/\"default-dockercfg-6wql8\"" Oct 31 19:56:10.281652 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.281406 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9902ffd9\"/\"openshift-service-ca.crt\"" Oct 31 19:56:10.281652 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.281461 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9902ffd9\"/\"e2e-lifecycle-model-9902ffd9-kserve-self-signed-certs\"" Oct 31 19:56:10.285103 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.285082 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf"] Oct 31 19:56:10.316317 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.316289 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-dshm\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.316433 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.316370 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tls-certs\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.316575 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.316548 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2rg\" (UniqueName: \"kubernetes.io/projected/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-kube-api-access-jj2rg\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.318669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.316603 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-model-cache\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.318669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.316696 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tmp-dir\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.318669 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.316790 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-home\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.421192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.418429 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tmp-dir\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.421192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.418568 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-home\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.421192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.418803 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-dshm\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.421192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.418897 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tls-certs\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.421192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.419081 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2rg\" (UniqueName: \"kubernetes.io/projected/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-kube-api-access-jj2rg\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.421192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.419140 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-model-cache\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.421192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.419717 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-model-cache\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.425037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.424142 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tmp-dir\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.425037 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.424605 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-home\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.428980 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.428951 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-dshm\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.432093 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.432018 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tls-certs\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.478822 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.478797 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2rg\" (UniqueName: \"kubernetes.io/projected/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-kube-api-access-jj2rg\") pod \"e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.496273 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.496252 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:10.915428 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:10.915397 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf"] Oct 31 19:56:10.916541 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:56:10.916513 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2561fe4_513a_4bdf_be15_68dfbbb8dcd0.slice/crio-5e08e9efc51926e0f9b3f9631847943f4c3e20d9b13865f04b813658046b27c3 WatchSource:0}: Error finding container 5e08e9efc51926e0f9b3f9631847943f4c3e20d9b13865f04b813658046b27c3: Status 404 returned error can't find the container with id 5e08e9efc51926e0f9b3f9631847943f4c3e20d9b13865f04b813658046b27c3 Oct 31 19:56:11.194250 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.194165 6481 generic.go:356] "Generic (PLEG): container finished" podID="dd23d352-acb4-4b63-addf-478cb67bc561" containerID="9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888" exitCode=0 Oct 31 19:56:11.194726 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.194239 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" event={"ID":"dd23d352-acb4-4b63-addf-478cb67bc561","Type":"ContainerDied","Data":"9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888"} Oct 31 19:56:11.194726 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.194663 6481 scope.go:122] "RemoveContainer" containerID="9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888" Oct 31 19:56:11.196056 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.196031 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" event={"ID":"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0","Type":"ContainerStarted","Data":"6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9"} Oct 31 19:56:11.196186 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.196062 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" event={"ID":"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0","Type":"ContainerStarted","Data":"5e08e9efc51926e0f9b3f9631847943f4c3e20d9b13865f04b813658046b27c3"} Oct 31 19:56:11.196473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.196445 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:11.196558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.196511 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:11.196558 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.196543 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:11.419458 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.419391 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" podStartSLOduration=1.419369602 podStartE2EDuration="1.419369602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:11.411270614 +0000 UTC m=+2342.651493745" watchObservedRunningTime="2026-10-31 19:56:11.419369602 +0000 UTC m=+2342.659592731" Oct 31 19:56:11.497234 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.497207 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:11.623054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.623017 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps"] Oct 31 19:56:11.743678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:11.741477 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp"] Oct 31 19:56:12.252925 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:12.252883 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" event={"ID":"dd23d352-acb4-4b63-addf-478cb67bc561","Type":"ContainerStarted","Data":"abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59"} Oct 31 19:56:12.891029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:12.889340 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:56:12.892163 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:12.892103 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 19:56:12.892288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:12.892165 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 19:56:13.887018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:13.886757 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 19:56:13.887018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:13.886831 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 19:56:14.891423 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:14.891382 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 19:56:14.891947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:14.891457 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 19:56:15.890489 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:15.890446 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 19:56:15.890713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:15.890528 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 19:56:16.564425 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:16.564392 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:56:16.623783 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:16.623741 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" podUID="cefbc76e-859f-4919-a607-2a75b90f3697" containerName="payload-processing" containerID="cri-o://6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06" gracePeriod=25 Oct 31 19:56:16.742413 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:16.742363 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" podUID="6936d404-a7af-48e0-ba93-96f4cff8e734" containerName="payload-pre-processing" containerID="cri-o://e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f" gracePeriod=25 Oct 31 19:56:16.895139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:16.894608 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 19:56:16.895139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:16.894671 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 19:56:16.946383 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:16.946357 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:56:17.098792 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.098762 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefbc76e-859f-4919-a607-2a75b90f3697-kube-api-access-xbl5f\" (UniqueName: \"kubernetes.io/projected/cefbc76e-859f-4919-a607-2a75b90f3697-kube-api-access-xbl5f\") pod \"cefbc76e-859f-4919-a607-2a75b90f3697\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " Oct 31 19:56:17.099326 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.099034 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefbc76e-859f-4919-a607-2a75b90f3697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbc76e-859f-4919-a607-2a75b90f3697-plugins-config-volume\") pod \"cefbc76e-859f-4919-a607-2a75b90f3697\" (UID: \"cefbc76e-859f-4919-a607-2a75b90f3697\") " Oct 31 19:56:17.099574 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.099455 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefbc76e-859f-4919-a607-2a75b90f3697-plugins-config-volume" pod "cefbc76e-859f-4919-a607-2a75b90f3697" (UID: "cefbc76e-859f-4919-a607-2a75b90f3697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:17.099734 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.099697 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbc76e-859f-4919-a607-2a75b90f3697-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:17.101657 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.101615 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefbc76e-859f-4919-a607-2a75b90f3697-kube-api-access-xbl5f" pod "cefbc76e-859f-4919-a607-2a75b90f3697" (UID: "cefbc76e-859f-4919-a607-2a75b90f3697"). InnerVolumeSpecName "kube-api-access-xbl5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:17.108288 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.108264 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:56:17.201841 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.201805 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbl5f\" (UniqueName: \"kubernetes.io/projected/cefbc76e-859f-4919-a607-2a75b90f3697-kube-api-access-xbl5f\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:17.303318 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.303281 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6936d404-a7af-48e0-ba93-96f4cff8e734-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936d404-a7af-48e0-ba93-96f4cff8e734-plugins-config-volume\") pod \"6936d404-a7af-48e0-ba93-96f4cff8e734\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " Oct 31 19:56:17.303491 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.303428 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6936d404-a7af-48e0-ba93-96f4cff8e734-kube-api-access-vdh22\" (UniqueName: \"kubernetes.io/projected/6936d404-a7af-48e0-ba93-96f4cff8e734-kube-api-access-vdh22\") pod \"6936d404-a7af-48e0-ba93-96f4cff8e734\" (UID: \"6936d404-a7af-48e0-ba93-96f4cff8e734\") " Oct 31 19:56:17.303740 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.303716 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6936d404-a7af-48e0-ba93-96f4cff8e734-plugins-config-volume" pod "6936d404-a7af-48e0-ba93-96f4cff8e734" (UID: "6936d404-a7af-48e0-ba93-96f4cff8e734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:17.304085 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.304069 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6936d404-a7af-48e0-ba93-96f4cff8e734-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:17.305516 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.305490 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6936d404-a7af-48e0-ba93-96f4cff8e734-kube-api-access-vdh22" pod "6936d404-a7af-48e0-ba93-96f4cff8e734" (UID: "6936d404-a7af-48e0-ba93-96f4cff8e734"). InnerVolumeSpecName "kube-api-access-vdh22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:17.316322 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.316293 6481 generic.go:356] "Generic (PLEG): container finished" podID="cefbc76e-859f-4919-a607-2a75b90f3697" containerID="6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06" exitCode=0 Oct 31 19:56:17.316441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.316335 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" Oct 31 19:56:17.316441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.316377 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" event={"ID":"cefbc76e-859f-4919-a607-2a75b90f3697","Type":"ContainerDied","Data":"6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06"} Oct 31 19:56:17.316441 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.316428 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps" event={"ID":"cefbc76e-859f-4919-a607-2a75b90f3697","Type":"ContainerDied","Data":"dce3a6b371f9b43ba6b87fe7e547c275ba2bb4809c6042909178bdd4477c7c72"} Oct 31 19:56:17.316590 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.316461 6481 scope.go:122] "RemoveContainer" containerID="6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06" Oct 31 19:56:17.317764 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.317739 6481 generic.go:356] "Generic (PLEG): container finished" podID="6936d404-a7af-48e0-ba93-96f4cff8e734" containerID="e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f" exitCode=0 Oct 31 19:56:17.317882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.317778 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" event={"ID":"6936d404-a7af-48e0-ba93-96f4cff8e734","Type":"ContainerDied","Data":"e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f"} Oct 31 19:56:17.317882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.317828 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" event={"ID":"6936d404-a7af-48e0-ba93-96f4cff8e734","Type":"ContainerDied","Data":"163272de163048c402e77b25b97cd0f760c38d6de5627fb344eba4f10def253d"} Oct 31 19:56:17.317882 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.317786 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp" Oct 31 19:56:17.328063 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.328045 6481 scope.go:122] "RemoveContainer" containerID="6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06" Oct 31 19:56:17.328382 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:17.328352 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06\": container with ID starting with 6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06 not found: ID does not exist" containerID="6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06" Oct 31 19:56:17.328477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.328392 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06"} err="failed to get container status \"6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06\": rpc error: code = NotFound desc = could not find container \"6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06\": container with ID starting with 6f0f8a527b137277bc8f930356163802bfb3016196683d7669596f2128e4ee06 not found: ID does not exist" Oct 31 19:56:17.328477 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.328414 6481 scope.go:122] "RemoveContainer" containerID="e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f" Oct 31 19:56:17.337547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.337526 6481 scope.go:122] "RemoveContainer" containerID="e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f" Oct 31 19:56:17.337789 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:17.337768 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f\": container with ID starting with e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f not found: ID does not exist" containerID="e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f" Oct 31 19:56:17.337851 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.337797 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f"} err="failed to get container status \"e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f\": rpc error: code = NotFound desc = could not find container \"e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f\": container with ID starting with e95226840ef652bf79d1158847b7bdc3a46eb792090c995d568e35d2063cd94f not found: ID does not exist" Oct 31 19:56:17.405308 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.405229 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdh22\" (UniqueName: \"kubernetes.io/projected/6936d404-a7af-48e0-ba93-96f4cff8e734-kube-api-access-vdh22\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:17.491104 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.491069 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp"] Oct 31 19:56:17.495375 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.495345 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f37f6f32-58dff996c5-t7bmp"] Oct 31 19:56:17.596837 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.596804 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps"] Oct 31 19:56:17.599750 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.599728 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f37f6f32-d4997d4f7-zrzps"] Oct 31 19:56:17.885423 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.885383 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 19:56:17.885611 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:17.885446 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 19:56:18.881460 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:18.881382 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6936d404-a7af-48e0-ba93-96f4cff8e734" path="/var/lib/kubelet/pods/6936d404-a7af-48e0-ba93-96f4cff8e734/volumes" Oct 31 19:56:18.881911 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:18.881894 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefbc76e-859f-4919-a607-2a75b90f3697" path="/var/lib/kubelet/pods/cefbc76e-859f-4919-a607-2a75b90f3697/volumes" Oct 31 19:56:18.889818 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:18.889797 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:56:18.890829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:18.890810 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 19:56:20.497071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:20.497039 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:20.509634 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:20.509608 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:21.358683 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:21.358655 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 19:56:23.109433 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.109397 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs"] Oct 31 19:56:23.110136 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.110121 6481 cpu_manager.go:414] "containerMap: removing container" podUID="cefbc76e-859f-4919-a607-2a75b90f3697" containerName="payload-processing" Oct 31 19:56:23.110189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.110137 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefbc76e-859f-4919-a607-2a75b90f3697" containerName="payload-processing" Oct 31 19:56:23.110189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.110172 6481 cpu_manager.go:414] "containerMap: removing container" podUID="6936d404-a7af-48e0-ba93-96f4cff8e734" containerName="payload-pre-processing" Oct 31 19:56:23.110189 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.110179 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6936d404-a7af-48e0-ba93-96f4cff8e734" containerName="payload-pre-processing" Oct 31 19:56:23.110304 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.110294 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefbc76e-859f-4919-a607-2a75b90f3697" containerName="payload-processing" Oct 31 19:56:23.110343 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.110310 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6936d404-a7af-48e0-ba93-96f4cff8e734" containerName="payload-pre-processing" Oct 31 19:56:23.114314 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.114296 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.209081 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.209046 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6eeb83-serving-cert\"" Oct 31 19:56:23.212786 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.212762 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs"] Oct 31 19:56:23.220466 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.220446 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da63ab8-4e17-473f-b41d-879a1e89350c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.220713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.220691 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.220757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.220743 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.220827 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.220814 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb84c\" (UniqueName: \"kubernetes.io/projected/9da63ab8-4e17-473f-b41d-879a1e89350c-kube-api-access-qb84c\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.322076 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.322038 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.322289 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.322092 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.322289 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.322111 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb84c\" (UniqueName: \"kubernetes.io/projected/9da63ab8-4e17-473f-b41d-879a1e89350c-kube-api-access-qb84c\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.322289 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.322171 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da63ab8-4e17-473f-b41d-879a1e89350c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.322289 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:23.322187 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-serving-cert: secret "maas-api-e2e-shared-b-w0-6eeb83-serving-cert" not found Oct 31 19:56:23.322289 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:23.322262 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls podName:9da63ab8-4e17-473f-b41d-879a1e89350c nodeName:}" failed. No retries permitted until 2026-10-31 19:56:23.822241724 +0000 UTC m=+2355.062464848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls") pod "maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" (UID: "9da63ab8-4e17-473f-b41d-879a1e89350c") : secret "maas-api-e2e-shared-b-w0-6eeb83-serving-cert" not found Oct 31 19:56:23.322875 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.322855 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da63ab8-4e17-473f-b41d-879a1e89350c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.324401 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.324382 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.414351 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.414271 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb84c\" (UniqueName: \"kubernetes.io/projected/9da63ab8-4e17-473f-b41d-879a1e89350c-kube-api-access-qb84c\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.841067 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.841029 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:23.843373 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:23.843351 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:24.028357 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:24.028318 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:24.398517 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:24.398489 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs"] Oct 31 19:56:24.400197 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:56:24.400169 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da63ab8_4e17_473f_b41d_879a1e89350c.slice/crio-0757074ab628c022155b0f920a43dda1483dcad89c5bebdd87610940a37f94fa WatchSource:0}: Error finding container 0757074ab628c022155b0f920a43dda1483dcad89c5bebdd87610940a37f94fa: Status 404 returned error can't find the container with id 0757074ab628c022155b0f920a43dda1483dcad89c5bebdd87610940a37f94fa Oct 31 19:56:24.423104 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:24.423068 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" event={"ID":"9da63ab8-4e17-473f-b41d-879a1e89350c","Type":"ContainerStarted","Data":"0757074ab628c022155b0f920a43dda1483dcad89c5bebdd87610940a37f94fa"} Oct 31 19:56:25.441819 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:25.441774 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" event={"ID":"9da63ab8-4e17-473f-b41d-879a1e89350c","Type":"ContainerStarted","Data":"1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c"} Oct 31 19:56:25.442402 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:25.442368 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:25.550988 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:25.550889 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" podStartSLOduration=1.691166044 podStartE2EDuration="2.550875368s" totalImagesPullingTime="859.709324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:25.548382529 +0000 UTC m=+2356.788605670" watchObservedRunningTime="2026-10-31 19:56:25.550875368 +0000 UTC m=+2356.791098496" Oct 31 19:56:27.326129 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:27.326093 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:56:32.464202 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:32.464170 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 19:56:42.359239 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:42.358449 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j"] Oct 31 19:56:42.360278 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:42.360234 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" podUID="7d7645d3-a774-4624-88dc-2b51b8a0f979" containerName="maas-api" containerID="cri-o://c62e17383dd720efed4af6d59009727dc01fe9e9504df0eb51137b983bb3f742" gracePeriod=30 Oct 31 19:56:42.661733 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:42.661630 6481 generic.go:356] "Generic (PLEG): container finished" podID="7d7645d3-a774-4624-88dc-2b51b8a0f979" containerID="c62e17383dd720efed4af6d59009727dc01fe9e9504df0eb51137b983bb3f742" exitCode=0 Oct 31 19:56:42.662149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:42.662117 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" event={"ID":"7d7645d3-a774-4624-88dc-2b51b8a0f979","Type":"ContainerDied","Data":"c62e17383dd720efed4af6d59009727dc01fe9e9504df0eb51137b983bb3f742"} Oct 31 19:56:43.056759 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.054809 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:56:43.155724 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.155695 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7645d3-a774-4624-88dc-2b51b8a0f979-kube-api-access-4gd2q\" (UniqueName: \"kubernetes.io/projected/7d7645d3-a774-4624-88dc-2b51b8a0f979-kube-api-access-4gd2q\") pod \"7d7645d3-a774-4624-88dc-2b51b8a0f979\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " Oct 31 19:56:43.155953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.155924 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7645d3-a774-4624-88dc-2b51b8a0f979-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7645d3-a774-4624-88dc-2b51b8a0f979-openshift-service-ca\") pod \"7d7645d3-a774-4624-88dc-2b51b8a0f979\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " Oct 31 19:56:43.156124 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.156084 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls\") pod \"7d7645d3-a774-4624-88dc-2b51b8a0f979\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " Oct 31 19:56:43.156181 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.156163 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-metrics-certs\") pod \"7d7645d3-a774-4624-88dc-2b51b8a0f979\" (UID: \"7d7645d3-a774-4624-88dc-2b51b8a0f979\") " Oct 31 19:56:43.156276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.156249 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7645d3-a774-4624-88dc-2b51b8a0f979-openshift-service-ca" pod "7d7645d3-a774-4624-88dc-2b51b8a0f979" (UID: "7d7645d3-a774-4624-88dc-2b51b8a0f979"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:43.156834 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.156812 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7645d3-a774-4624-88dc-2b51b8a0f979-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:43.157971 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.157939 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7645d3-a774-4624-88dc-2b51b8a0f979-kube-api-access-4gd2q" pod "7d7645d3-a774-4624-88dc-2b51b8a0f979" (UID: "7d7645d3-a774-4624-88dc-2b51b8a0f979"). InnerVolumeSpecName "kube-api-access-4gd2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:43.158235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.158205 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls" pod "7d7645d3-a774-4624-88dc-2b51b8a0f979" (UID: "7d7645d3-a774-4624-88dc-2b51b8a0f979"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:56:43.158235 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.158215 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-metrics-certs" pod "7d7645d3-a774-4624-88dc-2b51b8a0f979" (UID: "7d7645d3-a774-4624-88dc-2b51b8a0f979"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:56:43.257964 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.257877 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:43.257964 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.257908 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7645d3-a774-4624-88dc-2b51b8a0f979-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:43.257964 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.257918 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gd2q\" (UniqueName: \"kubernetes.io/projected/7d7645d3-a774-4624-88dc-2b51b8a0f979-kube-api-access-4gd2q\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:43.670215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.670128 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" Oct 31 19:56:43.670215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.670142 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j" event={"ID":"7d7645d3-a774-4624-88dc-2b51b8a0f979","Type":"ContainerDied","Data":"32b4936c81e5c876e25e49b831b1760dbdf6daa6cae28eb85412062bb348392e"} Oct 31 19:56:43.670215 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.670213 6481 scope.go:122] "RemoveContainer" containerID="c62e17383dd720efed4af6d59009727dc01fe9e9504df0eb51137b983bb3f742" Oct 31 19:56:43.827759 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.827726 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j"] Oct 31 19:56:43.874005 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:43.873966 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f37f6f32-f79886d8f-hn45j"] Oct 31 19:56:44.116126 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.116094 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn"] Oct 31 19:56:44.116870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.116844 6481 cpu_manager.go:414] "containerMap: removing container" podUID="7d7645d3-a774-4624-88dc-2b51b8a0f979" containerName="maas-api" Oct 31 19:56:44.116870 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.116860 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7645d3-a774-4624-88dc-2b51b8a0f979" containerName="maas-api" Oct 31 19:56:44.117025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.117016 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7645d3-a774-4624-88dc-2b51b8a0f979" containerName="maas-api" Oct 31 19:56:44.122208 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.122191 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.170333 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.170305 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5370f5-serving-cert\"" Oct 31 19:56:44.173309 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.173276 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn"] Oct 31 19:56:44.176816 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.176795 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9c28a9-2f92-4649-b02b-da2425aa3f72-openshift-service-ca\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.176933 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.176853 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.176984 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.176939 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-metrics-certs\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.177116 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.177095 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsfbd\" (UniqueName: \"kubernetes.io/projected/4b9c28a9-2f92-4649-b02b-da2425aa3f72-kube-api-access-zsfbd\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.278546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.278507 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9c28a9-2f92-4649-b02b-da2425aa3f72-openshift-service-ca\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.278746 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.278589 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.278746 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.278612 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-metrics-certs\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.278746 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.278724 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsfbd\" (UniqueName: \"kubernetes.io/projected/4b9c28a9-2f92-4649-b02b-da2425aa3f72-kube-api-access-zsfbd\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.278921 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:44.278740 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-serving-cert: secret "maas-api-e2e-api-a-5370f5-serving-cert" not found Oct 31 19:56:44.278921 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:44.278824 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls podName:4b9c28a9-2f92-4649-b02b-da2425aa3f72 nodeName:}" failed. No retries permitted until 2026-10-31 19:56:44.778793892 +0000 UTC m=+2376.019017015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls") pod "maas-api-e2e-api-a-5370f5-f87784475-l5rgn" (UID: "4b9c28a9-2f92-4649-b02b-da2425aa3f72") : secret "maas-api-e2e-api-a-5370f5-serving-cert" not found Oct 31 19:56:44.279331 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.279311 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9c28a9-2f92-4649-b02b-da2425aa3f72-openshift-service-ca\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.281204 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.281181 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-metrics-certs\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.332579 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.332545 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsfbd\" (UniqueName: \"kubernetes.io/projected/4b9c28a9-2f92-4649-b02b-da2425aa3f72-kube-api-access-zsfbd\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.814449 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.814405 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.817030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.816968 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls\") pod \"maas-api-e2e-api-a-5370f5-f87784475-l5rgn\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:44.880891 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:44.880858 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7645d3-a774-4624-88dc-2b51b8a0f979" path="/var/lib/kubelet/pods/7d7645d3-a774-4624-88dc-2b51b8a0f979/volumes" Oct 31 19:56:45.036027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:45.035979 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:45.407628 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:45.407591 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn"] Oct 31 19:56:45.409201 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:56:45.409173 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9c28a9_2f92_4649_b02b_da2425aa3f72.slice/crio-899126f297bce07e47ab8b29985afdc8fcc0f94a6e9517e38002d68d500e569a WatchSource:0}: Error finding container 899126f297bce07e47ab8b29985afdc8fcc0f94a6e9517e38002d68d500e569a: Status 404 returned error can't find the container with id 899126f297bce07e47ab8b29985afdc8fcc0f94a6e9517e38002d68d500e569a Oct 31 19:56:45.726914 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:45.726880 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" event={"ID":"4b9c28a9-2f92-4649-b02b-da2425aa3f72","Type":"ContainerStarted","Data":"899126f297bce07e47ab8b29985afdc8fcc0f94a6e9517e38002d68d500e569a"} Oct 31 19:56:46.735945 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:46.735900 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" event={"ID":"4b9c28a9-2f92-4649-b02b-da2425aa3f72","Type":"ContainerStarted","Data":"bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159"} Oct 31 19:56:46.736376 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:46.735967 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:46.845354 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:46.845307 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" podStartSLOduration=1.9325500230000001 podStartE2EDuration="2.845294427s" totalImagesPullingTime="912.744404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:46.843212782 +0000 UTC m=+2378.083435913" watchObservedRunningTime="2026-10-31 19:56:46.845294427 +0000 UTC m=+2378.085517556" Oct 31 19:56:47.139070 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.130661 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr"] Oct 31 19:56:47.199554 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.199524 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.230019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.211507 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr"] Oct 31 19:56:47.296430 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.293480 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-ddb5a8-openshift-default-dockercfg-2hct4\"" Oct 31 19:56:47.320953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.256958 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-credential-socket\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.320953 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.320899 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istiod-ca-cert\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.321231 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.320963 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9cm\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-kube-api-access-fh9cm\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.321231 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.320988 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-podinfo\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.326800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.321413 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-token\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.326800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.321465 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-envoy\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.326800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.321511 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-socket\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.326800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.321577 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-certs\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.326800 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.321605 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-data\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.423321 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.423180 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9cm\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-kube-api-access-fh9cm\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.425046 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.425009 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-podinfo\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.435168 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.435080 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-token\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.435706 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-envoy\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.435748 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-socket\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.435794 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-certs\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.435816 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-data\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.435861 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-credential-socket\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436039 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.435900 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istiod-ca-cert\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436886 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.436739 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istiod-ca-cert\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.436886 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.436768 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-certs\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.437063 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.437045 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-data\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.444071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.437289 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-credential-socket\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.444071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.442349 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-socket\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.444071 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.442613 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-podinfo\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.495043 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.494958 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-envoy\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.498324 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.498267 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-token\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.504271 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.504236 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9cm\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-kube-api-access-fh9cm\") pod \"e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:47.647083 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:47.646207 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:48.148320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:48.148288 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr"] Oct 31 19:56:48.149631 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:56:48.149605 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf1f6bb_a7b0_41d0_a01d_0a988af14ea9.slice/crio-f0beaec6771d174ba4bf8c831aefb265ebbef75ebd0156e64c26b67c75cb4922 WatchSource:0}: Error finding container f0beaec6771d174ba4bf8c831aefb265ebbef75ebd0156e64c26b67c75cb4922: Status 404 returned error can't find the container with id f0beaec6771d174ba4bf8c831aefb265ebbef75ebd0156e64c26b67c75cb4922 Oct 31 19:56:48.152632 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:48.152597 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:48.152739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:48.152673 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:48.152739 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:48.152715 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:56:48.769481 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:48.769444 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" event={"ID":"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9","Type":"ContainerStarted","Data":"eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93"} Oct 31 19:56:48.769655 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:48.769526 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" event={"ID":"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9","Type":"ContainerStarted","Data":"f0beaec6771d174ba4bf8c831aefb265ebbef75ebd0156e64c26b67c75cb4922"} Oct 31 19:56:49.646969 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:49.646936 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:49.652888 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:49.652867 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:49.776956 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:49.776922 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:49.777753 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:49.777732 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:56:49.781111 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:49.781060 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podStartSLOduration=2.7810351559999997 podStartE2EDuration="2.781035156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:48.891552545 +0000 UTC m=+2380.131775691" watchObservedRunningTime="2026-10-31 19:56:49.781035156 +0000 UTC m=+2381.021258287" Oct 31 19:56:52.768761 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:52.768729 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 19:56:53.139601 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:53.139520 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l"] Oct 31 19:56:53.139758 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:53.139727 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" containerID="cri-o://1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4" gracePeriod=30 Oct 31 19:56:53.827320 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:53.827289 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7"] Oct 31 19:56:53.827803 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:53.827519 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" containerID="cri-o://508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84" gracePeriod=30 Oct 31 19:56:57.384025 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:57.382736 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:56:58.448657 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.448625 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:56:58.550502 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550470 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-credential-socket\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550700 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550554 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-envoy\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550700 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550593 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-certs\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550700 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550619 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-token\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550700 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550651 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-podinfo\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550700 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550672 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istiod-ca-cert\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550712 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-kube-api-access-rrbgx\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-kube-api-access-rrbgx\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550768 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-data\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.550975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550776 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-credential-socket" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.550975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.550894 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-socket\") pod \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\" (UID: \"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397\") " Oct 31 19:56:58.551212 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551067 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-certs" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.551274 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551211 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-data" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.551328 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551293 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istiod-ca-cert" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:58.551417 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551380 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-socket" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.551947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551922 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.552077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551953 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.552077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551967 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.552077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.551981 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.552077 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.552013 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.553047 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.553018 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-envoy" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:58.553169 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.553082 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-podinfo" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:58.553169 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.553141 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-kube-api-access-rrbgx" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "kube-api-access-rrbgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:58.553546 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.553527 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-token" pod "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" (UID: "bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:58.654217 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.654180 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.654217 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.654218 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.654422 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.654233 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.654422 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.654247 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrbgx\" (UniqueName: \"kubernetes.io/projected/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397-kube-api-access-rrbgx\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:58.888935 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.888904 6481 generic.go:356] "Generic (PLEG): container finished" podID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerID="1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4" exitCode=0 Oct 31 19:56:58.889135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.888948 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" Oct 31 19:56:58.889135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.888962 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" event={"ID":"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397","Type":"ContainerDied","Data":"1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4"} Oct 31 19:56:58.889135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.889055 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l" event={"ID":"bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397","Type":"ContainerDied","Data":"8d4dd3f31f5bbba8a5b64dfaac530b2fb8151d11fe8e9e83999419f569d50803"} Oct 31 19:56:58.889135 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.889088 6481 scope.go:122] "RemoveContainer" containerID="1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4" Oct 31 19:56:58.940789 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.940762 6481 scope.go:122] "RemoveContainer" containerID="5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf" Oct 31 19:56:58.950928 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.950904 6481 scope.go:122] "RemoveContainer" containerID="1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4" Oct 31 19:56:58.951223 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:58.951204 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4\": container with ID starting with 1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4 not found: ID does not exist" containerID="1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4" Oct 31 19:56:58.951295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.951232 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4"} err="failed to get container status \"1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4\": rpc error: code = NotFound desc = could not find container \"1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4\": container with ID starting with 1a75753309232192a1f5a617408e7cc327da49fe2402b74ff2aaf57544c8a5a4 not found: ID does not exist" Oct 31 19:56:58.951295 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.951248 6481 scope.go:122] "RemoveContainer" containerID="5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf" Oct 31 19:56:58.951543 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:58.951522 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf\": container with ID starting with 5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf not found: ID does not exist" containerID="5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf" Oct 31 19:56:58.951605 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:58.951550 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf"} err="failed to get container status \"5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf\": rpc error: code = NotFound desc = could not find container \"5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf\": container with ID starting with 5902f399f734c1999e7cdcba0288afe243abdeb99008db32abce22345a14afbf not found: ID does not exist" Oct 31 19:56:59.046809 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.046762 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l"] Oct 31 19:56:59.050027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.049988 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f37f6f32-openshift-default-c5f8cf77c-wj49l"] Oct 31 19:56:59.087899 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.087877 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:56:59.160682 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.160599 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-kube-api-access-d4r7r\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-kube-api-access-d4r7r\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.160854 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.160684 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-token\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-token\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.160854 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.160801 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-certs\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.160854 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.160838 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-socket\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.161059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.160876 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-envoy\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.161059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.160936 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68856c11-a0fe-4119-9f1d-256ab6271cdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68856c11-a0fe-4119-9f1d-256ab6271cdb-istiod-ca-cert\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.161059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.160974 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-podinfo\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.161225 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161061 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-credential-socket\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.161225 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161102 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-data\") pod \"68856c11-a0fe-4119-9f1d-256ab6271cdb\" (UID: \"68856c11-a0fe-4119-9f1d-256ab6271cdb\") " Oct 31 19:56:59.161225 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161104 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-certs" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:59.161391 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161228 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-socket" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:59.161451 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161386 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-data" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:59.161504 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161469 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-credential-socket" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:59.161735 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161708 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68856c11-a0fe-4119-9f1d-256ab6271cdb-istiod-ca-cert" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:59.161950 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161931 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.162059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161955 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.162059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161971 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68856c11-a0fe-4119-9f1d-256ab6271cdb-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.162059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.161986 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.162059 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.162037 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.163133 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.163107 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-token" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:59.163399 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.163382 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-envoy" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:59.163633 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.163609 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-podinfo" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:59.163691 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.163636 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-kube-api-access-d4r7r" pod "68856c11-a0fe-4119-9f1d-256ab6271cdb" (UID: "68856c11-a0fe-4119-9f1d-256ab6271cdb"). InnerVolumeSpecName "kube-api-access-d4r7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:59.263172 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.263138 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4r7r\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-kube-api-access-d4r7r\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.263172 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.263166 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.263172 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.263176 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.263421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.263186 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68856c11-a0fe-4119-9f1d-256ab6271cdb-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:59.897457 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.897423 6481 generic.go:356] "Generic (PLEG): container finished" podID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerID="508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84" exitCode=0 Oct 31 19:56:59.897457 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.897457 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" Oct 31 19:56:59.898029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.897508 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" event={"ID":"68856c11-a0fe-4119-9f1d-256ab6271cdb","Type":"ContainerDied","Data":"508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84"} Oct 31 19:56:59.898029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.897563 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7" event={"ID":"68856c11-a0fe-4119-9f1d-256ab6271cdb","Type":"ContainerDied","Data":"d121150ce11b6d45d66ce70be82ff7402af7fa7880482b2151d9a36f2ca1a0af"} Oct 31 19:56:59.898029 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.897594 6481 scope.go:122] "RemoveContainer" containerID="508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84" Oct 31 19:56:59.908116 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.908099 6481 scope.go:122] "RemoveContainer" containerID="508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84" Oct 31 19:56:59.908380 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:56:59.908354 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84\": container with ID starting with 508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84 not found: ID does not exist" containerID="508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84" Oct 31 19:56:59.908428 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:56:59.908378 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84"} err="failed to get container status \"508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84\": rpc error: code = NotFound desc = could not find container \"508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84\": container with ID starting with 508637d15b3262927ef838cd02b8c38b59c57eec90870f7b4594bf30ab8abd84 not found: ID does not exist" Oct 31 19:57:00.127018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:00.120672 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7"] Oct 31 19:57:00.177899 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:00.177809 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-th4d7"] Oct 31 19:57:00.905163 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:00.904229 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" path="/var/lib/kubelet/pods/68856c11-a0fe-4119-9f1d-256ab6271cdb/volumes" Oct 31 19:57:00.905667 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:00.905266 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" path="/var/lib/kubelet/pods/bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397/volumes" Oct 31 19:57:05.351864 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.351825 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm"] Oct 31 19:57:05.352723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.352698 6481 cpu_manager.go:414] "containerMap: removing container" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" Oct 31 19:57:05.352723 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.352723 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" Oct 31 19:57:05.352900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.352780 6481 cpu_manager.go:414] "containerMap: removing container" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" Oct 31 19:57:05.352900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.352789 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" Oct 31 19:57:05.352900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.352823 6481 cpu_manager.go:414] "containerMap: removing container" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" Oct 31 19:57:05.352900 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.352830 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" Oct 31 19:57:05.353139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.353018 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="68856c11-a0fe-4119-9f1d-256ab6271cdb" containerName="istio-proxy" Oct 31 19:57:05.353139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.353037 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" Oct 31 19:57:05.353139 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.353056 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9dba8e-c1d3-4fd4-bb6b-d1a7312cb397" containerName="istio-proxy" Oct 31 19:57:05.356711 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.356689 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.455028 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.451509 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-650cf3-openshift-default-dockercfg-fzdnn\"" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681252 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-certs\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681346 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827ffe64-1796-44a5-b134-99bdca43134f-istio-podinfo\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681398 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-credential-socket\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681544 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-socket\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681591 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-istio-token\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681632 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dzk\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-kube-api-access-j8dzk\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681781 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-envoy\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.681884 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827ffe64-1796-44a5-b134-99bdca43134f-istiod-ca-cert\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.691149 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.682213 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-data\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:05.765020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:05.746965 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm"] Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.130431 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-socket\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.130490 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-istio-token\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.130527 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dzk\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-kube-api-access-j8dzk\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.130607 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-envoy\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.130718 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827ffe64-1796-44a5-b134-99bdca43134f-istiod-ca-cert\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.131007 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-data\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.131130 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-certs\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.131178 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827ffe64-1796-44a5-b134-99bdca43134f-istio-podinfo\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.136785 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.131219 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-credential-socket\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.232023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.231666 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827ffe64-1796-44a5-b134-99bdca43134f-istiod-ca-cert\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.271017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.267871 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-data\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.271017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.268218 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-certs\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.318647 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.318606 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-socket\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.319171 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.319146 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-credential-socket\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.338034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.335527 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-envoy\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.338034 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.336702 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827ffe64-1796-44a5-b134-99bdca43134f-istio-podinfo\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.379025 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.378873 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-istio-token\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.391398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.391326 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dzk\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-kube-api-access-j8dzk\") pod \"e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.484519 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.484471 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:57:06.510349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.485798 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:06.571023 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:06.565183 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:57:07.517904 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:57:07.517872 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827ffe64_1796_44a5_b134_99bdca43134f.slice/crio-5c705adce8acaa7e6818f0b38304d02b06e24d68c517863b9fb779d766f229d1 WatchSource:0}: Error finding container 5c705adce8acaa7e6818f0b38304d02b06e24d68c517863b9fb779d766f229d1: Status 404 returned error can't find the container with id 5c705adce8acaa7e6818f0b38304d02b06e24d68c517863b9fb779d766f229d1 Oct 31 19:57:07.525108 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:07.524979 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm"] Oct 31 19:57:07.527704 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:07.527668 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:07.527780 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:07.527749 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:07.527815 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:07.527791 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:08.248018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:08.247971 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" event={"ID":"827ffe64-1796-44a5-b134-99bdca43134f","Type":"ContainerStarted","Data":"2f3753804e83e1e94847988f0647b8159f377fa17e86df84e28be820f584788c"} Oct 31 19:57:08.248279 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:08.248258 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" event={"ID":"827ffe64-1796-44a5-b134-99bdca43134f","Type":"ContainerStarted","Data":"5c705adce8acaa7e6818f0b38304d02b06e24d68c517863b9fb779d766f229d1"} Oct 31 19:57:08.428027 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:08.425039 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" podStartSLOduration=3.424984157 podStartE2EDuration="3.424984157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:08.363406144 +0000 UTC m=+2399.603629322" watchObservedRunningTime="2026-10-31 19:57:08.424984157 +0000 UTC m=+2399.665207287" Oct 31 19:57:08.490017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:08.489905 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:08.581349 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:08.581269 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:09.258562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:09.258379 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:09.259935 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:09.259912 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:57:09.961192 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:09.961156 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:57:09.961685 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:09.961258 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:57:14.679152 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:14.679117 6481 patch_prober.go:29] interesting pod/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:57:15.615575 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:14.679193 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:57:16.346934 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.346907 6481 generic.go:356] "Generic (PLEG): container finished" podID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerID="243fce928ba1b7c0ddaada721e4139d003fc6d568ff8ac19fe10626a8a0779b7" exitCode=0 Oct 31 19:57:16.348152 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.346984 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" event={"ID":"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5","Type":"ContainerDied","Data":"243fce928ba1b7c0ddaada721e4139d003fc6d568ff8ac19fe10626a8a0779b7"} Oct 31 19:57:16.348152 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.347553 6481 scope.go:122] "RemoveContainer" containerID="243fce928ba1b7c0ddaada721e4139d003fc6d568ff8ac19fe10626a8a0779b7" Oct 31 19:57:16.350031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.350006 6481 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerID="87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32" exitCode=0 Oct 31 19:57:16.350162 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.350025 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" event={"ID":"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520","Type":"ContainerDied","Data":"87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32"} Oct 31 19:57:16.350352 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.350320 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:16.350421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.350408 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:16.350478 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.350452 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:16.350531 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.350474 6481 scope.go:122] "RemoveContainer" containerID="87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32" Oct 31 19:57:16.352436 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.352406 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:16.352542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.352484 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:16.352542 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:16.352525 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:17.359276 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.359239 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" event={"ID":"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5","Type":"ContainerStarted","Data":"21a0f438b96955476b0867592b092fd13bab50cc82c55c1a7bdea926e68e685a"} Oct 31 19:57:17.361470 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.361439 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" event={"ID":"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520","Type":"ContainerStarted","Data":"efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885"} Oct 31 19:57:17.363057 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.362989 6481 generic.go:356] "Generic (PLEG): container finished" podID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerID="f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678" exitCode=0 Oct 31 19:57:17.363166 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.363037 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" event={"ID":"5ee90507-77f4-497c-b6b6-70b47a953dec","Type":"ContainerDied","Data":"f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678"} Oct 31 19:57:17.363416 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.363403 6481 scope.go:122] "RemoveContainer" containerID="f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678" Oct 31 19:57:17.364899 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.364876 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:17.364965 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.364938 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:17.365082 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.364969 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:17.836012 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.831737 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:57:17.836012 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.831927 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 19:57:17.836012 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.831974 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 19:57:17.924227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.924095 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:57:17.924227 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.924209 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:57:17.924486 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:17.924260 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 19:57:18.390856 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:18.390818 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" event={"ID":"5ee90507-77f4-497c-b6b6-70b47a953dec","Type":"ContainerStarted","Data":"b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5"} Oct 31 19:57:18.769387 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:18.769349 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 19:57:18.769689 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:18.769657 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 19:57:18.938698 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:18.938649 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:57:18.939069 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:18.939037 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 19:57:19.301020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:19.284396 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:57:19.301020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:19.284530 6481 patch_prober.go:29] interesting pod/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 31 19:57:19.301020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:19.284579 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 31 19:57:19.783015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:19.769454 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 19:57:19.783015 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:19.769530 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 19:57:19.935019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:19.934386 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:57:19.935019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:19.934453 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 19:57:20.287019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:20.286252 6481 patch_prober.go:29] interesting pod/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 31 19:57:20.287019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:20.286323 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 31 19:57:20.772020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:20.769404 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 19:57:20.772020 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:20.769476 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 19:57:20.944113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:20.933379 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:57:20.944113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:20.933440 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 19:57:21.283451 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:21.283411 6481 patch_prober.go:29] interesting pod/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 31 19:57:21.284105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:21.284074 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 31 19:57:21.769021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:21.766372 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 19:57:21.769021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:21.766444 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 19:57:21.926019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:21.925613 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:57:21.926019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:21.925699 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 19:57:22.287017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:22.286433 6481 patch_prober.go:29] interesting pod/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 31 19:57:22.287017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:22.286496 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 31 19:57:22.771019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:22.769555 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 31 19:57:22.771019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:22.769616 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 31 19:57:22.926022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:22.924470 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:57:22.926022 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:22.924552 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 31 19:57:23.286018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.285412 6481 patch_prober.go:29] interesting pod/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 31 19:57:23.286018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.285494 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 31 19:57:23.795355 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.795323 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:57:23.795760 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.795723 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:57:23.796455 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.796436 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 19:57:23.929379 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.929347 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:57:23.929675 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.929652 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:57:23.930262 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:23.930242 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 19:57:24.283819 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.283780 6481 patch_prober.go:29] interesting pod/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 31 19:57:24.284216 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.284177 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 31 19:57:24.395474 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.395440 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn"] Oct 31 19:57:24.400072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.400048 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.448947 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.448919 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-9a8346-serving-cert\"" Oct 31 19:57:24.453972 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.453946 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn"] Oct 31 19:57:24.507198 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.507155 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.507434 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.507404 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42k5q\" (UniqueName: \"kubernetes.io/projected/6103cd8a-4d26-40d7-b08c-e76492bd3d15-kube-api-access-42k5q\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.507709 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.507672 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.507891 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.507866 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6103cd8a-4d26-40d7-b08c-e76492bd3d15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.609705 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.609620 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.609705 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.609674 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6103cd8a-4d26-40d7-b08c-e76492bd3d15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.609894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.609757 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.609894 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.609844 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42k5q\" (UniqueName: \"kubernetes.io/projected/6103cd8a-4d26-40d7-b08c-e76492bd3d15-kube-api-access-42k5q\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.609960 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:57:24.609926 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-serving-cert: secret "maas-api-e2e-shared-b-w4-9a8346-serving-cert" not found Oct 31 19:57:24.610025 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:57:24.610012 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls podName:6103cd8a-4d26-40d7-b08c-e76492bd3d15 nodeName:}" failed. No retries permitted until 2026-10-31 19:57:25.109978947 +0000 UTC m=+2416.350202075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls") pod "maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" (UID: "6103cd8a-4d26-40d7-b08c-e76492bd3d15") : secret "maas-api-e2e-shared-b-w4-9a8346-serving-cert" not found Oct 31 19:57:24.610586 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.610559 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6103cd8a-4d26-40d7-b08c-e76492bd3d15-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.612140 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.612120 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:24.695017 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:24.685215 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42k5q\" (UniqueName: \"kubernetes.io/projected/6103cd8a-4d26-40d7-b08c-e76492bd3d15-kube-api-access-42k5q\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:25.126119 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.126085 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:25.128449 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.128422 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:25.288012 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.287973 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:57:25.288418 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.288398 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:57:25.288972 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.288949 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 19:57:25.314463 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.314437 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:25.689462 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.689432 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn"] Oct 31 19:57:25.690586 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:57:25.690558 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6103cd8a_4d26_40d7_b08c_e76492bd3d15.slice/crio-cac6b08c5d71a09145d46e274abcf464c53891be62a0b01d720e4b3be21d8231 WatchSource:0}: Error finding container cac6b08c5d71a09145d46e274abcf464c53891be62a0b01d720e4b3be21d8231: Status 404 returned error can't find the container with id cac6b08c5d71a09145d46e274abcf464c53891be62a0b01d720e4b3be21d8231 Oct 31 19:57:25.692259 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:25.692237 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:57:26.487833 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:26.487793 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" event={"ID":"6103cd8a-4d26-40d7-b08c-e76492bd3d15","Type":"ContainerStarted","Data":"cac6b08c5d71a09145d46e274abcf464c53891be62a0b01d720e4b3be21d8231"} Oct 31 19:57:27.326222 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:57:27.326185 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:57:27.496138 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:27.496099 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" event={"ID":"6103cd8a-4d26-40d7-b08c-e76492bd3d15","Type":"ContainerStarted","Data":"1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3"} Oct 31 19:57:27.496573 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:27.496161 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:27.638072 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:27.636988 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" podStartSLOduration=2.725552341 podStartE2EDuration="3.6369673s" totalImagesPullingTime="911.414959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:27.620327585 +0000 UTC m=+2418.860550714" watchObservedRunningTime="2026-10-31 19:57:27.6369673 +0000 UTC m=+2418.877190430" Oct 31 19:57:33.507394 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:33.507361 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 19:57:55.760975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:55.760939 6481 generic.go:356] "Generic (PLEG): container finished" podID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerID="0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb" exitCode=0 Oct 31 19:57:55.761422 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:55.761024 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" event={"ID":"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e","Type":"ContainerDied","Data":"0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb"} Oct 31 19:57:55.761501 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:55.761485 6481 scope.go:122] "RemoveContainer" containerID="0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb" Oct 31 19:57:55.763236 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:55.763207 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:55.763301 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:55.763279 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:55.763337 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:55.763309 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:57:56.794031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:56.793962 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" event={"ID":"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e","Type":"ContainerStarted","Data":"80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0"} Oct 31 19:57:56.908738 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:56.908708 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:57:56.908931 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:56.908898 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 19:57:56.909035 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:56.908937 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 19:57:57.326856 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:57:57.326819 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:57:57.908562 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:57.908521 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 19:57:57.909089 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:57.908589 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 19:57:58.908581 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:58.908546 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 19:57:58.908958 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:58.908603 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 19:57:59.908906 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:59.908867 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 19:57:59.909437 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:57:59.908936 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 19:58:00.908658 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:00.908625 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 19:58:00.908840 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:00.908688 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 19:58:01.750644 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:01.750614 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7"] Oct 31 19:58:01.754720 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:01.754691 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:01.843506 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:01.843477 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dfe658-openshift-default-dockercfg-dzpbl\"" Oct 31 19:58:01.862593 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:01.862561 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7"] Oct 31 19:58:02.123576 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.123470 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 31 19:58:02.125247 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.123618 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 31 19:58:02.145901 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.145365 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:58:02.158108 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.158069 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istiod-ca-cert\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.158450 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.158417 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-socket\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.158631 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.158614 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-data\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.158832 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.158800 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-token\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.159098 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.159079 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-podinfo\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.159265 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.159252 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-certs\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.159406 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.159391 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlck5\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-kube-api-access-nlck5\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.159595 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.159568 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-envoy\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.159747 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.159723 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-credential-socket\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.261473 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.261429 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-podinfo\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.262549 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.262521 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-certs\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.262778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.262762 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlck5\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-kube-api-access-nlck5\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.262988 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.262967 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-envoy\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.263171 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.263153 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-credential-socket\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.264101 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.264074 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-certs\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.264210 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.263869 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-credential-socket\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.264321 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.264297 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istiod-ca-cert\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.264421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.264406 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-socket\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.264515 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.264494 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-data\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.264583 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.264534 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-token\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.264685 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.264668 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-socket\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.265319 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.265295 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-data\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.265445 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.265424 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istiod-ca-cert\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.266195 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.266169 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-podinfo\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.268208 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.268170 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-envoy\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.315865 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.315822 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlck5\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-kube-api-access-nlck5\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.317079 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.317055 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-token\") pod \"e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.376806 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.376721 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:02.804526 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.804496 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7"] Oct 31 19:58:02.805578 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:58:02.805554 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9fa3be_2c9a_4960_a69d_fb1e8adcbfbc.slice/crio-ffe589b2789a0c0f5550af3bbb34b91fc9e94c5c2e19e364ade8110da148778c WatchSource:0}: Error finding container ffe589b2789a0c0f5550af3bbb34b91fc9e94c5c2e19e364ade8110da148778c: Status 404 returned error can't find the container with id ffe589b2789a0c0f5550af3bbb34b91fc9e94c5c2e19e364ade8110da148778c Oct 31 19:58:02.808377 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.808341 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:58:02.808488 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.808408 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:58:02.808488 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.808441 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:58:02.913489 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.913462 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:58:02.923817 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.923779 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" event={"ID":"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc","Type":"ContainerStarted","Data":"22c2f61f4dbd721813592284365975197739d928393ff61cccf10e3d3eb3c47d"} Oct 31 19:58:02.924021 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.923841 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" event={"ID":"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc","Type":"ContainerStarted","Data":"ffe589b2789a0c0f5550af3bbb34b91fc9e94c5c2e19e364ade8110da148778c"} Oct 31 19:58:02.924807 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:02.924782 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 19:58:03.132201 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:03.132096 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podStartSLOduration=2.132075892 podStartE2EDuration="2.132075892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:03.130751847 +0000 UTC m=+2454.370974969" watchObservedRunningTime="2026-10-31 19:58:03.132075892 +0000 UTC m=+2454.372299023" Oct 31 19:58:03.384246 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:03.381374 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:03.396421 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:03.396384 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:03.930763 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:03.930730 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:03.931713 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:03.931695 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 19:58:20.406358 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.406317 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp"] Oct 31 19:58:20.409775 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.409759 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.506670 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.505827 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-ddb5a8-serving-cert\"" Oct 31 19:58:20.514031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.512651 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp"] Oct 31 19:58:20.514031 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.513943 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd3a97-c608-434c-a37f-bfbc79806c2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.514253 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.514171 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.514315 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.514252 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.514569 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.514537 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswr7\" (UniqueName: \"kubernetes.io/projected/5bcd3a97-c608-434c-a37f-bfbc79806c2f-kube-api-access-bswr7\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.645121 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.645079 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswr7\" (UniqueName: \"kubernetes.io/projected/5bcd3a97-c608-434c-a37f-bfbc79806c2f-kube-api-access-bswr7\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.658975 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.658879 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd3a97-c608-434c-a37f-bfbc79806c2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.659539 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.659508 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.661808 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.661756 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.662433 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:58:20.662412 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-serving-cert: secret "maas-api-e2e-shared-b-w6-ddb5a8-serving-cert" not found Oct 31 19:58:20.662720 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:58:20.662695 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls podName:5bcd3a97-c608-434c-a37f-bfbc79806c2f nodeName:}" failed. No retries permitted until 2026-10-31 19:58:21.162671099 +0000 UTC m=+2472.402894226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls") pod "maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" (UID: "5bcd3a97-c608-434c-a37f-bfbc79806c2f") : secret "maas-api-e2e-shared-b-w6-ddb5a8-serving-cert" not found Oct 31 19:58:20.663086 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.660793 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd3a97-c608-434c-a37f-bfbc79806c2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.668829 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.668800 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:20.701891 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:20.701858 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswr7\" (UniqueName: \"kubernetes.io/projected/5bcd3a97-c608-434c-a37f-bfbc79806c2f-kube-api-access-bswr7\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:21.189018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:21.185242 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:21.205339 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:21.200573 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:21.323475 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:21.323446 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:21.696118 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:21.696093 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp"] Oct 31 19:58:21.698459 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:58:21.698427 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcd3a97_c608_434c_a37f_bfbc79806c2f.slice/crio-ed050a27d5e2cf3a0486c8ee75c06b1f2e96a7ae95c827c14a9a14369b896abc WatchSource:0}: Error finding container ed050a27d5e2cf3a0486c8ee75c06b1f2e96a7ae95c827c14a9a14369b896abc: Status 404 returned error can't find the container with id ed050a27d5e2cf3a0486c8ee75c06b1f2e96a7ae95c827c14a9a14369b896abc Oct 31 19:58:22.110541 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:22.110502 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" event={"ID":"5bcd3a97-c608-434c-a37f-bfbc79806c2f","Type":"ContainerStarted","Data":"ed050a27d5e2cf3a0486c8ee75c06b1f2e96a7ae95c827c14a9a14369b896abc"} Oct 31 19:58:23.155244 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:23.155199 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" event={"ID":"5bcd3a97-c608-434c-a37f-bfbc79806c2f","Type":"ContainerStarted","Data":"7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8"} Oct 31 19:58:23.155778 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:23.155303 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:23.265113 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:23.265062 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" podStartSLOduration=2.327760181 podStartE2EDuration="3.265046296s" totalImagesPullingTime="937.286115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:23.261458822 +0000 UTC m=+2474.501681952" watchObservedRunningTime="2026-10-31 19:58:23.265046296 +0000 UTC m=+2474.505269425" Oct 31 19:58:27.328809 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:58:27.328773 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:58:29.167922 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:58:29.167895 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 19:58:57.326124 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:58:57.326084 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:59:02.543909 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:02.543866 6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerID="eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93" exitCode=0 Oct 31 19:59:02.544529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:02.543941 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" event={"ID":"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9","Type":"ContainerDied","Data":"eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93"} Oct 31 19:59:02.544600 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:02.544540 6481 scope.go:122] "RemoveContainer" containerID="eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93" Oct 31 19:59:02.546364 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:02.546336 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:59:02.546443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:02.546408 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:59:02.546489 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:02.546443 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:59:03.099042 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.099006 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11074-b5cc64"] Oct 31 19:59:03.102573 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.102554 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:03.152815 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.152783 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xptzx\"" Oct 31 19:59:03.155615 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.155584 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11074-b5cc64"] Oct 31 19:59:03.190630 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.190595 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8n4s\" (UniqueName: \"kubernetes.io/projected/5ec00e11-5f4e-47a7-880d-2703d72a6892-kube-api-access-w8n4s\") pod \"test-curl-11074-b5cc64\" (UID: \"5ec00e11-5f4e-47a7-880d-2703d72a6892\") " pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:03.292104 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.292073 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8n4s\" (UniqueName: \"kubernetes.io/projected/5ec00e11-5f4e-47a7-880d-2703d72a6892-kube-api-access-w8n4s\") pod \"test-curl-11074-b5cc64\" (UID: \"5ec00e11-5f4e-47a7-880d-2703d72a6892\") " pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:03.346820 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.346780 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8n4s\" (UniqueName: \"kubernetes.io/projected/5ec00e11-5f4e-47a7-880d-2703d72a6892-kube-api-access-w8n4s\") pod \"test-curl-11074-b5cc64\" (UID: \"5ec00e11-5f4e-47a7-880d-2703d72a6892\") " pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:03.414768 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.414688 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:03.557129 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.557074 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" event={"ID":"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9","Type":"ContainerStarted","Data":"87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02"} Oct 31 19:59:03.647389 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.647354 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:59:03.648398 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.648365 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 19:59:03.648524 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.648418 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 19:59:03.827666 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:03.827638 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11074-b5cc64"] Oct 31 19:59:03.829641 ip-10-0-143-134 kubenswrapper[6481]: W1031 19:59:03.829615 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ec00e11_5f4e_47a7_880d_2703d72a6892.slice/crio-fece66fa9b572eee5cb4c403dd882f1de8b37be00bbd245a73351fc3443466f6 WatchSource:0}: Error finding container fece66fa9b572eee5cb4c403dd882f1de8b37be00bbd245a73351fc3443466f6: Status 404 returned error can't find the container with id fece66fa9b572eee5cb4c403dd882f1de8b37be00bbd245a73351fc3443466f6 Oct 31 19:59:04.584678 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:04.584642 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11074-b5cc64" event={"ID":"5ec00e11-5f4e-47a7-880d-2703d72a6892","Type":"ContainerStarted","Data":"4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002"} Oct 31 19:59:04.594258 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:04.592106 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11074-b5cc64" event={"ID":"5ec00e11-5f4e-47a7-880d-2703d72a6892","Type":"ContainerStarted","Data":"fece66fa9b572eee5cb4c403dd882f1de8b37be00bbd245a73351fc3443466f6"} Oct 31 19:59:04.646929 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:04.646902 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 19:59:04.647112 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:04.646955 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 19:59:04.693247 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:04.693201 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11074-b5cc64" podStartSLOduration=1.6931857 podStartE2EDuration="1.6931857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:04.690269858 +0000 UTC m=+2515.930492988" watchObservedRunningTime="2026-10-31 19:59:04.6931857 +0000 UTC m=+2515.933408829" Oct 31 19:59:04.777605 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:04.777568 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:59:05.676019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:05.659626 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 19:59:05.676019 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:05.659698 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 19:59:06.215286 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:06.215254 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11074-b5cc64"] Oct 31 19:59:06.220305 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:06.220276 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11074-b5cc64"] Oct 31 19:59:06.629988 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:06.629725 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11074-b5cc64" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" containerName="test-curl-11074-b5cc64" containerID="cri-o://4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002" gracePeriod=2 Oct 31 19:59:06.647754 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:06.647724 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 19:59:06.647885 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:06.647778 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 19:59:06.690259 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:59:06.690219 6481 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11074-b5cc64\" is forbidden: User \"system:node:ip-10-0-143-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-west-2.compute.internal' and this object" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:07.647587 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:07.647549 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 19:59:07.647766 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:07.647613 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 19:59:08.647206 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:08.647128 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 19:59:08.647757 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:08.647224 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 19:59:08.927779 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:08.927753 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:08.983721 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:59:08.983673 6481 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11074-b5cc64\" is forbidden: User \"system:node:ip-10-0-143-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-west-2.compute.internal' and this object" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:08.993501 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:08.993467 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec00e11-5f4e-47a7-880d-2703d72a6892-kube-api-access-w8n4s\" (UniqueName: \"kubernetes.io/projected/5ec00e11-5f4e-47a7-880d-2703d72a6892-kube-api-access-w8n4s\") pod \"5ec00e11-5f4e-47a7-880d-2703d72a6892\" (UID: \"5ec00e11-5f4e-47a7-880d-2703d72a6892\") " Oct 31 19:59:08.996062 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:08.996034 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec00e11-5f4e-47a7-880d-2703d72a6892-kube-api-access-w8n4s" pod "5ec00e11-5f4e-47a7-880d-2703d72a6892" (UID: "5ec00e11-5f4e-47a7-880d-2703d72a6892"). InnerVolumeSpecName "kube-api-access-w8n4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:09.096214 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.096172 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8n4s\" (UniqueName: \"kubernetes.io/projected/5ec00e11-5f4e-47a7-880d-2703d72a6892-kube-api-access-w8n4s\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.574584 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.574546 6481 scope.go:122] "RemoveContainer" containerID="21050ce929ea535ce699f0437774fe18b818bd5774e6844ae76517b2c5185b40" Oct 31 19:59:09.652924 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.652896 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:59:09.654054 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.654029 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 19:59:09.657918 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.657893 6481 generic.go:356] "Generic (PLEG): container finished" podID="5ec00e11-5f4e-47a7-880d-2703d72a6892" containerID="4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002" exitCode=137 Oct 31 19:59:09.658030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.657929 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:09.658030 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.657989 6481 scope.go:122] "RemoveContainer" containerID="4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002" Oct 31 19:59:09.668431 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.668414 6481 scope.go:122] "RemoveContainer" containerID="4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002" Oct 31 19:59:09.668708 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:59:09.668686 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002\": container with ID starting with 4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002 not found: ID does not exist" containerID="4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002" Oct 31 19:59:09.668776 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:09.668714 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002"} err="failed to get container status \"4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002\": rpc error: code = NotFound desc = could not find container \"4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002\": container with ID starting with 4abd02ae701a2a535332ea97a5f344c792187218c9688481922f4e62b01e9002 not found: ID does not exist" Oct 31 19:59:09.855598 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:59:09.855500 6481 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11074-b5cc64\" is forbidden: User \"system:node:ip-10-0-143-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-west-2.compute.internal' and this object" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:10.013411 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:59:10.013366 6481 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11074-b5cc64\" is forbidden: User \"system:node:ip-10-0-143-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-west-2.compute.internal' and this object" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" pod="opendatahub/test-curl-11074-b5cc64" Oct 31 19:59:10.883248 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:10.883191 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" path="/var/lib/kubelet/pods/5ec00e11-5f4e-47a7-880d-2703d72a6892/volumes" Oct 31 19:59:27.326180 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:59:27.326101 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 19:59:47.140105 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:47.140072 6481 generic.go:356] "Generic (PLEG): container finished" podID="827ffe64-1796-44a5-b134-99bdca43134f" containerID="2f3753804e83e1e94847988f0647b8159f377fa17e86df84e28be820f584788c" exitCode=0 Oct 31 19:59:47.140547 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:47.140154 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" event={"ID":"827ffe64-1796-44a5-b134-99bdca43134f","Type":"ContainerDied","Data":"2f3753804e83e1e94847988f0647b8159f377fa17e86df84e28be820f584788c"} Oct 31 19:59:47.140677 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:47.140659 6481 scope.go:122] "RemoveContainer" containerID="2f3753804e83e1e94847988f0647b8159f377fa17e86df84e28be820f584788c" Oct 31 19:59:47.142461 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:47.142437 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:59:47.142529 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:47.142510 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:59:47.142568 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:47.142553 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 19:59:48.149268 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:48.149227 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" event={"ID":"827ffe64-1796-44a5-b134-99bdca43134f","Type":"ContainerStarted","Data":"b199b2b494f2b40c1f4ebf5082016155d67c1cc304713627e5034c79523bf1d9"} Oct 31 19:59:48.487823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:48.487351 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:59:48.487823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:48.487483 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 31 19:59:48.487823 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:48.487540 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 31 19:59:49.490443 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:49.490403 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 31 19:59:49.490986 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:49.490489 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 31 19:59:50.486190 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:50.486149 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 31 19:59:50.486472 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:50.486444 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 31 19:59:51.486826 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:51.486785 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 31 19:59:51.487463 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:51.487431 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 31 19:59:52.489018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:52.488672 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 31 19:59:52.489018 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:52.488770 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 31 19:59:53.491771 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:53.491739 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:59:53.492241 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:53.492224 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:59:53.492764 ip-10-0-143-134 kubenswrapper[6481]: I1031 19:59:53.492745 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 19:59:57.326481 ip-10-0-143-134 kubenswrapper[6481]: E1031 19:59:57.326443 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:00:02.335253 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:02.335215 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt"] Oct 31 20:00:02.456811 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:02.456779 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j"] Oct 31 20:00:07.335975 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:07.335934 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" podUID="56c22de4-a8ca-42ea-991b-857382cba34a" containerName="payload-processing" containerID="cri-o://98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6" gracePeriod=25 Oct 31 20:00:07.457227 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:07.457197 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" podUID="37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" containerName="payload-pre-processing" containerID="cri-o://9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c" gracePeriod=25 Oct 31 20:00:07.716366 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:07.716331 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 20:00:07.825241 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:07.824061 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c22de4-a8ca-42ea-991b-857382cba34a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c22de4-a8ca-42ea-991b-857382cba34a-plugins-config-volume\") pod \"56c22de4-a8ca-42ea-991b-857382cba34a\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " Oct 31 20:00:07.825241 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:07.824269 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c22de4-a8ca-42ea-991b-857382cba34a-kube-api-access-zvwfk\" (UniqueName: \"kubernetes.io/projected/56c22de4-a8ca-42ea-991b-857382cba34a-kube-api-access-zvwfk\") pod \"56c22de4-a8ca-42ea-991b-857382cba34a\" (UID: \"56c22de4-a8ca-42ea-991b-857382cba34a\") " Oct 31 20:00:07.826604 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:07.826539 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c22de4-a8ca-42ea-991b-857382cba34a-plugins-config-volume" pod "56c22de4-a8ca-42ea-991b-857382cba34a" (UID: "56c22de4-a8ca-42ea-991b-857382cba34a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:07.947889 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:07.941197 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c22de4-a8ca-42ea-991b-857382cba34a-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.028851 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.028797 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c22de4-a8ca-42ea-991b-857382cba34a-kube-api-access-zvwfk" pod "56c22de4-a8ca-42ea-991b-857382cba34a" (UID: "56c22de4-a8ca-42ea-991b-857382cba34a"). InnerVolumeSpecName "kube-api-access-zvwfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:08.043508 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.042610 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwfk\" (UniqueName: \"kubernetes.io/projected/56c22de4-a8ca-42ea-991b-857382cba34a-kube-api-access-zvwfk\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.050966 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.050936 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 20:00:08.244571 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.244541 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-kube-api-access-blkls\" (UniqueName: \"kubernetes.io/projected/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-kube-api-access-blkls\") pod \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " Oct 31 20:00:08.244745 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.244596 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-plugins-config-volume\") pod \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\" (UID: \"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf\") " Oct 31 20:00:08.245020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.244979 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-plugins-config-volume" pod "37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" (UID: "37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:08.247108 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.247072 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-kube-api-access-blkls" pod "37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" (UID: "37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf"). InnerVolumeSpecName "kube-api-access-blkls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:08.328600 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.328563 6481 generic.go:356] "Generic (PLEG): container finished" podID="56c22de4-a8ca-42ea-991b-857382cba34a" containerID="98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6" exitCode=0 Oct 31 20:00:08.328791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.328610 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" Oct 31 20:00:08.328791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.328649 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" event={"ID":"56c22de4-a8ca-42ea-991b-857382cba34a","Type":"ContainerDied","Data":"98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6"} Oct 31 20:00:08.328791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.328711 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt" event={"ID":"56c22de4-a8ca-42ea-991b-857382cba34a","Type":"ContainerDied","Data":"4ea8627aed1377ce90ee7e5e9ea7264fa9d390a02c1399fd5ca0446dd51d3fef"} Oct 31 20:00:08.328791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.328739 6481 scope.go:122] "RemoveContainer" containerID="98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6" Oct 31 20:00:08.330412 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.330386 6481 generic.go:356] "Generic (PLEG): container finished" podID="37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" containerID="9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c" exitCode=0 Oct 31 20:00:08.330596 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.330576 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" event={"ID":"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf","Type":"ContainerDied","Data":"9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c"} Oct 31 20:00:08.330676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.330620 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" event={"ID":"37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf","Type":"ContainerDied","Data":"6fb13c2926d63c66c485bf64bc038d64cc86be0779a1ad44da9f3173a1acd8a4"} Oct 31 20:00:08.330763 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.330696 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j" Oct 31 20:00:08.339541 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.339314 6481 scope.go:122] "RemoveContainer" containerID="98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6" Oct 31 20:00:08.339818 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:00:08.339602 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6\": container with ID starting with 98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6 not found: ID does not exist" containerID="98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6" Oct 31 20:00:08.339818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.339632 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6"} err="failed to get container status \"98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6\": rpc error: code = NotFound desc = could not find container \"98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6\": container with ID starting with 98d19fe5049e92ea26be7cdb082c7fd11392179451eec5d26aa0cec5f37c93e6 not found: ID does not exist" Oct 31 20:00:08.339818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.339656 6481 scope.go:122] "RemoveContainer" containerID="9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c" Oct 31 20:00:08.346097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.346074 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkls\" (UniqueName: \"kubernetes.io/projected/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-kube-api-access-blkls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.346179 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.346098 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.391114 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.391078 6481 scope.go:122] "RemoveContainer" containerID="9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c" Oct 31 20:00:08.391504 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:00:08.391474 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c\": container with ID starting with 9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c not found: ID does not exist" containerID="9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c" Oct 31 20:00:08.391669 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.391634 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c"} err="failed to get container status \"9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c\": rpc error: code = NotFound desc = could not find container \"9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c\": container with ID starting with 9ca12cc922d7ecb7eb6812e4c8d8167fa60cc98b081f69b4a93de8718f30c12c not found: ID does not exist" Oct 31 20:00:08.498059 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.497960 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt"] Oct 31 20:00:08.502353 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.502322 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a5ba62-86f8c9999f-qmmgt"] Oct 31 20:00:08.604244 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.604215 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j"] Oct 31 20:00:08.611324 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.611298 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a5ba62-76b8dd88dc-smq5j"] Oct 31 20:00:08.910059 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.909963 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" path="/var/lib/kubelet/pods/37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf/volumes" Oct 31 20:00:08.910598 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:08.910572 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c22de4-a8ca-42ea-991b-857382cba34a" path="/var/lib/kubelet/pods/56c22de4-a8ca-42ea-991b-857382cba34a/volumes" Oct 31 20:00:14.376491 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:14.376456 6481 generic.go:356] "Generic (PLEG): container finished" podID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerID="22c2f61f4dbd721813592284365975197739d928393ff61cccf10e3d3eb3c47d" exitCode=0 Oct 31 20:00:14.376922 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:14.376531 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" event={"ID":"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc","Type":"ContainerDied","Data":"22c2f61f4dbd721813592284365975197739d928393ff61cccf10e3d3eb3c47d"} Oct 31 20:00:14.377044 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:14.376988 6481 scope.go:122] "RemoveContainer" containerID="22c2f61f4dbd721813592284365975197739d928393ff61cccf10e3d3eb3c47d" Oct 31 20:00:14.378668 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:14.378639 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:00:14.378763 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:14.378729 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:00:14.378763 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:14.378760 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:00:15.426019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:15.413147 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" event={"ID":"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc","Type":"ContainerStarted","Data":"ffe3c9e0f098d15d503900db1adf7910db78a16d41d7aa399a2b853f122441d0"} Oct 31 20:00:16.385018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:16.384053 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 20:00:16.390017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:16.388094 6481 patch_prober.go:29] interesting pod/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:00:16.390017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:16.388166 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 20:00:17.387398 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:17.387352 6481 patch_prober.go:29] interesting pod/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:00:17.387950 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:17.387433 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 20:00:18.382014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:18.377352 6481 patch_prober.go:29] interesting pod/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:00:18.382014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:18.377424 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 20:00:18.950015 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:18.944090 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 20:00:19.392017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:19.387794 6481 patch_prober.go:29] interesting pod/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:00:19.392017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:19.387883 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 20:00:20.390406 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:20.390375 6481 patch_prober.go:29] interesting pod/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 20:00:20.390936 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:20.390484 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 20:00:21.382219 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:21.382190 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 20:00:21.383320 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:21.383295 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 20:00:23.252752 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:23.252715 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg"] Oct 31 20:00:23.371458 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:23.371421 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s"] Oct 31 20:00:27.325352 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:00:27.325310 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:00:28.283152 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.283100 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" podUID="7532c752-6a84-43a7-ba46-b6c38c03f8c7" containerName="payload-processing" containerID="cri-o://2a881f68dbd789f2082daaa642a223edf72bb73a7e1e05a8174d52857a74ff19" gracePeriod=25 Oct 31 20:00:28.399024 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.390330 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" podUID="ca110c79-031f-4725-8471-6dc641713674" containerName="payload-pre-processing" containerID="cri-o://66a8a1d7257b3a91783518c936b6dcd1fa2918475a74038df16545a114f72fff" gracePeriod=25 Oct 31 20:00:28.565927 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.565773 6481 generic.go:356] "Generic (PLEG): container finished" podID="7532c752-6a84-43a7-ba46-b6c38c03f8c7" containerID="2a881f68dbd789f2082daaa642a223edf72bb73a7e1e05a8174d52857a74ff19" exitCode=0 Oct 31 20:00:28.565927 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.565863 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" event={"ID":"7532c752-6a84-43a7-ba46-b6c38c03f8c7","Type":"ContainerDied","Data":"2a881f68dbd789f2082daaa642a223edf72bb73a7e1e05a8174d52857a74ff19"} Oct 31 20:00:28.568397 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.568370 6481 generic.go:356] "Generic (PLEG): container finished" podID="ca110c79-031f-4725-8471-6dc641713674" containerID="66a8a1d7257b3a91783518c936b6dcd1fa2918475a74038df16545a114f72fff" exitCode=0 Oct 31 20:00:28.568563 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.568543 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" event={"ID":"ca110c79-031f-4725-8471-6dc641713674","Type":"ContainerDied","Data":"66a8a1d7257b3a91783518c936b6dcd1fa2918475a74038df16545a114f72fff"} Oct 31 20:00:28.701144 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.701119 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 20:00:28.720706 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.720685 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 20:00:28.840692 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.840620 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca110c79-031f-4725-8471-6dc641713674-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca110c79-031f-4725-8471-6dc641713674-plugins-config-volume\") pod \"ca110c79-031f-4725-8471-6dc641713674\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " Oct 31 20:00:28.840854 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.840712 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca110c79-031f-4725-8471-6dc641713674-kube-api-access-xjr59\" (UniqueName: \"kubernetes.io/projected/ca110c79-031f-4725-8471-6dc641713674-kube-api-access-xjr59\") pod \"ca110c79-031f-4725-8471-6dc641713674\" (UID: \"ca110c79-031f-4725-8471-6dc641713674\") " Oct 31 20:00:28.840854 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.840764 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7532c752-6a84-43a7-ba46-b6c38c03f8c7-kube-api-access-qhhpl\" (UniqueName: \"kubernetes.io/projected/7532c752-6a84-43a7-ba46-b6c38c03f8c7-kube-api-access-qhhpl\") pod \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " Oct 31 20:00:28.841091 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.841065 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca110c79-031f-4725-8471-6dc641713674-plugins-config-volume" pod "ca110c79-031f-4725-8471-6dc641713674" (UID: "ca110c79-031f-4725-8471-6dc641713674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:28.841296 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.841274 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7532c752-6a84-43a7-ba46-b6c38c03f8c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7532c752-6a84-43a7-ba46-b6c38c03f8c7-plugins-config-volume\") pod \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\" (UID: \"7532c752-6a84-43a7-ba46-b6c38c03f8c7\") " Oct 31 20:00:28.841377 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.841319 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7532c752-6a84-43a7-ba46-b6c38c03f8c7-plugins-config-volume" pod "7532c752-6a84-43a7-ba46-b6c38c03f8c7" (UID: "7532c752-6a84-43a7-ba46-b6c38c03f8c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:28.841941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.841921 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7532c752-6a84-43a7-ba46-b6c38c03f8c7-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:28.841941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.841941 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca110c79-031f-4725-8471-6dc641713674-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:28.842949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.842930 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca110c79-031f-4725-8471-6dc641713674-kube-api-access-xjr59" pod "ca110c79-031f-4725-8471-6dc641713674" (UID: "ca110c79-031f-4725-8471-6dc641713674"). InnerVolumeSpecName "kube-api-access-xjr59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:28.843319 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.843305 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7532c752-6a84-43a7-ba46-b6c38c03f8c7-kube-api-access-qhhpl" pod "7532c752-6a84-43a7-ba46-b6c38c03f8c7" (UID: "7532c752-6a84-43a7-ba46-b6c38c03f8c7"). InnerVolumeSpecName "kube-api-access-qhhpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:28.943726 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.943690 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjr59\" (UniqueName: \"kubernetes.io/projected/ca110c79-031f-4725-8471-6dc641713674-kube-api-access-xjr59\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:28.943726 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:28.943718 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhhpl\" (UniqueName: \"kubernetes.io/projected/7532c752-6a84-43a7-ba46-b6c38c03f8c7-kube-api-access-qhhpl\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:29.576446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.576415 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" Oct 31 20:00:29.576446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.576429 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s" event={"ID":"ca110c79-031f-4725-8471-6dc641713674","Type":"ContainerDied","Data":"cd3dcf8fbcce2cc8187fd4097bd5b366476e14c73712dd32058d9a539e8b0dec"} Oct 31 20:00:29.576950 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.576489 6481 scope.go:122] "RemoveContainer" containerID="66a8a1d7257b3a91783518c936b6dcd1fa2918475a74038df16545a114f72fff" Oct 31 20:00:29.578338 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.578310 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" event={"ID":"7532c752-6a84-43a7-ba46-b6c38c03f8c7","Type":"ContainerDied","Data":"51a21341d89a70fa0aae5cf83550f3776e210cdca8b6bf9a1efe69d4cedff2df"} Oct 31 20:00:29.578467 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.578339 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg" Oct 31 20:00:29.586944 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.586927 6481 scope.go:122] "RemoveContainer" containerID="2a881f68dbd789f2082daaa642a223edf72bb73a7e1e05a8174d52857a74ff19" Oct 31 20:00:29.730954 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.730920 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg"] Oct 31 20:00:29.778924 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.778889 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1de410-577f8f8f57-bwrwg"] Oct 31 20:00:29.835327 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.835296 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s"] Oct 31 20:00:29.840961 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:29.840899 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1de410-6dfc49c7f9-jdz7s"] Oct 31 20:00:30.882058 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:30.882020 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7532c752-6a84-43a7-ba46-b6c38c03f8c7" path="/var/lib/kubelet/pods/7532c752-6a84-43a7-ba46-b6c38c03f8c7/volumes" Oct 31 20:00:30.882703 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:30.882663 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca110c79-031f-4725-8471-6dc641713674" path="/var/lib/kubelet/pods/ca110c79-031f-4725-8471-6dc641713674/volumes" Oct 31 20:00:35.513416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:35.513380 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5"] Oct 31 20:00:35.514558 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:35.514525 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" podUID="6d113153-5532-437d-b16b-c30fa0c8721f" containerName="maas-api" containerID="cri-o://95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5" gracePeriod=30 Oct 31 20:00:36.152078 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.152054 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 20:00:36.216848 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.216815 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls\") pod \"6d113153-5532-437d-b16b-c30fa0c8721f\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " Oct 31 20:00:36.217034 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.216892 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d113153-5532-437d-b16b-c30fa0c8721f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d113153-5532-437d-b16b-c30fa0c8721f-openshift-service-ca\") pod \"6d113153-5532-437d-b16b-c30fa0c8721f\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " Oct 31 20:00:36.217123 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.217104 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-metrics-certs\") pod \"6d113153-5532-437d-b16b-c30fa0c8721f\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " Oct 31 20:00:36.217229 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.217206 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d113153-5532-437d-b16b-c30fa0c8721f-kube-api-access-6nf2l\" (UniqueName: \"kubernetes.io/projected/6d113153-5532-437d-b16b-c30fa0c8721f-kube-api-access-6nf2l\") pod \"6d113153-5532-437d-b16b-c30fa0c8721f\" (UID: \"6d113153-5532-437d-b16b-c30fa0c8721f\") " Oct 31 20:00:36.217378 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.217349 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d113153-5532-437d-b16b-c30fa0c8721f-openshift-service-ca" pod "6d113153-5532-437d-b16b-c30fa0c8721f" (UID: "6d113153-5532-437d-b16b-c30fa0c8721f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:36.218036 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.217978 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d113153-5532-437d-b16b-c30fa0c8721f-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:36.219353 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.219320 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls" pod "6d113153-5532-437d-b16b-c30fa0c8721f" (UID: "6d113153-5532-437d-b16b-c30fa0c8721f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:36.219717 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.219691 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d113153-5532-437d-b16b-c30fa0c8721f-kube-api-access-6nf2l" pod "6d113153-5532-437d-b16b-c30fa0c8721f" (UID: "6d113153-5532-437d-b16b-c30fa0c8721f"). InnerVolumeSpecName "kube-api-access-6nf2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:36.219812 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.219767 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-metrics-certs" pod "6d113153-5532-437d-b16b-c30fa0c8721f" (UID: "6d113153-5532-437d-b16b-c30fa0c8721f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:36.319519 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.319486 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:36.319519 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.319516 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nf2l\" (UniqueName: \"kubernetes.io/projected/6d113153-5532-437d-b16b-c30fa0c8721f-kube-api-access-6nf2l\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:36.319519 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.319526 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d113153-5532-437d-b16b-c30fa0c8721f-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:36.751017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.745337 6481 generic.go:356] "Generic (PLEG): container finished" podID="6d113153-5532-437d-b16b-c30fa0c8721f" containerID="95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5" exitCode=0 Oct 31 20:00:36.751017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.745457 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" event={"ID":"6d113153-5532-437d-b16b-c30fa0c8721f","Type":"ContainerDied","Data":"95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5"} Oct 31 20:00:36.751017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.745514 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" event={"ID":"6d113153-5532-437d-b16b-c30fa0c8721f","Type":"ContainerDied","Data":"4ebf8611daf0edbd6cd124a44fbf0a729189045baa2f8dcb68ce82cbc66834a6"} Oct 31 20:00:36.751017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.745551 6481 scope.go:122] "RemoveContainer" containerID="95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5" Oct 31 20:00:36.751017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.745751 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5" Oct 31 20:00:36.792051 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.791120 6481 scope.go:122] "RemoveContainer" containerID="95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5" Oct 31 20:00:36.793523 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:00:36.792652 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5\": container with ID starting with 95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5 not found: ID does not exist" containerID="95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5" Oct 31 20:00:36.793523 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.792704 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5"} err="failed to get container status \"95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5\": rpc error: code = NotFound desc = could not find container \"95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5\": container with ID starting with 95daf2d6cf0771fbac2ff92ad19163295ca570bc29aca6e71b9f6bcd8af917c5 not found: ID does not exist" Oct 31 20:00:36.912358 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.912324 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5"] Oct 31 20:00:36.958317 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:36.958289 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7c0d34-dfb6c7b69-g26l5"] Oct 31 20:00:38.880373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:00:38.880341 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d113153-5532-437d-b16b-c30fa0c8721f" path="/var/lib/kubelet/pods/6d113153-5532-437d-b16b-c30fa0c8721f/volumes" Oct 31 20:00:57.326366 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:00:57.326281 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:01:20.544848 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:20.544808 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9"] Oct 31 20:01:20.545446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:20.545070 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" podUID="798bcb87-4cb1-4aac-8e6c-04033dff6888" containerName="maas-api" containerID="cri-o://5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a" gracePeriod=30 Oct 31 20:01:20.893215 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:20.893190 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 20:01:21.044683 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.044643 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798bcb87-4cb1-4aac-8e6c-04033dff6888-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798bcb87-4cb1-4aac-8e6c-04033dff6888-openshift-service-ca\") pod \"798bcb87-4cb1-4aac-8e6c-04033dff6888\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " Oct 31 20:01:21.055707 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.055675 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-metrics-certs\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-metrics-certs\") pod \"798bcb87-4cb1-4aac-8e6c-04033dff6888\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " Oct 31 20:01:21.056534 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.056511 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798bcb87-4cb1-4aac-8e6c-04033dff6888-kube-api-access-vr7m4\" (UniqueName: \"kubernetes.io/projected/798bcb87-4cb1-4aac-8e6c-04033dff6888-kube-api-access-vr7m4\") pod \"798bcb87-4cb1-4aac-8e6c-04033dff6888\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " Oct 31 20:01:21.056817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.056801 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls\") pod \"798bcb87-4cb1-4aac-8e6c-04033dff6888\" (UID: \"798bcb87-4cb1-4aac-8e6c-04033dff6888\") " Oct 31 20:01:21.074059 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.053213 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798bcb87-4cb1-4aac-8e6c-04033dff6888-openshift-service-ca" pod "798bcb87-4cb1-4aac-8e6c-04033dff6888" (UID: "798bcb87-4cb1-4aac-8e6c-04033dff6888"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:21.075806 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.065625 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-metrics-certs" pod "798bcb87-4cb1-4aac-8e6c-04033dff6888" (UID: "798bcb87-4cb1-4aac-8e6c-04033dff6888"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:21.077521 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.075157 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls" pod "798bcb87-4cb1-4aac-8e6c-04033dff6888" (UID: "798bcb87-4cb1-4aac-8e6c-04033dff6888"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:21.077682 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.075757 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798bcb87-4cb1-4aac-8e6c-04033dff6888-kube-api-access-vr7m4" pod "798bcb87-4cb1-4aac-8e6c-04033dff6888" (UID: "798bcb87-4cb1-4aac-8e6c-04033dff6888"). InnerVolumeSpecName "kube-api-access-vr7m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:21.160235 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.159966 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798bcb87-4cb1-4aac-8e6c-04033dff6888-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:21.160478 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.160456 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:21.160580 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.160566 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr7m4\" (UniqueName: \"kubernetes.io/projected/798bcb87-4cb1-4aac-8e6c-04033dff6888-kube-api-access-vr7m4\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:21.160761 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.160742 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798bcb87-4cb1-4aac-8e6c-04033dff6888-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:21.262710 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.262659 6481 generic.go:356] "Generic (PLEG): container finished" podID="798bcb87-4cb1-4aac-8e6c-04033dff6888" containerID="5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a" exitCode=0 Oct 31 20:01:21.262886 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.262722 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" Oct 31 20:01:21.262886 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.262734 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" event={"ID":"798bcb87-4cb1-4aac-8e6c-04033dff6888","Type":"ContainerDied","Data":"5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a"} Oct 31 20:01:21.262886 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.262776 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9" event={"ID":"798bcb87-4cb1-4aac-8e6c-04033dff6888","Type":"ContainerDied","Data":"2438551fa366937b3cb9ad4f8b413db1c785c673bb75ac7ed6fa47acb717997c"} Oct 31 20:01:21.262886 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.262800 6481 scope.go:122] "RemoveContainer" containerID="5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a" Oct 31 20:01:21.273519 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.273499 6481 scope.go:122] "RemoveContainer" containerID="5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a" Oct 31 20:01:21.273772 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:21.273752 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a\": container with ID starting with 5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a not found: ID does not exist" containerID="5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a" Oct 31 20:01:21.273814 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.273782 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a"} err="failed to get container status \"5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a\": rpc error: code = NotFound desc = could not find container \"5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a\": container with ID starting with 5c0fb96033b0d37eb2e89c76a1e4b9c8f463c32009b8ae182d572cab2856db7a not found: ID does not exist" Oct 31 20:01:21.416428 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.416345 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9"] Oct 31 20:01:21.462108 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:21.462077 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a5ba62-fc48bdd77-8cbt9"] Oct 31 20:01:22.880565 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:22.880535 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798bcb87-4cb1-4aac-8e6c-04033dff6888" path="/var/lib/kubelet/pods/798bcb87-4cb1-4aac-8e6c-04033dff6888/volumes" Oct 31 20:01:27.325490 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:27.325454 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:01:29.631499 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:29.631464 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f"] Oct 31 20:01:29.631941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:29.631673 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" containerID="cri-o://abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59" gracePeriod=30 Oct 31 20:01:33.893217 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:33.893181 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:01:33.893603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:33.893236 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:01:34.904217 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.904161 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 20:01:34.990761 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.990730 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-kube-api-access-r6zz7\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-kube-api-access-r6zz7\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.990928 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.990796 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-certs\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.990928 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.990844 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-socket\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.991089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991074 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-istio-token\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-istio-token\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.991154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991131 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-envoy\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.991154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991070 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-certs" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:34.991154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991138 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-socket" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:34.991323 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991261 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd23d352-acb4-4b63-addf-478cb67bc561-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd23d352-acb4-4b63-addf-478cb67bc561-istiod-ca-cert\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.991323 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991301 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-data\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.991438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991338 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-credential-socket\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.991438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991365 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd23d352-acb4-4b63-addf-478cb67bc561-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd23d352-acb4-4b63-addf-478cb67bc561-istio-podinfo\") pod \"dd23d352-acb4-4b63-addf-478cb67bc561\" (UID: \"dd23d352-acb4-4b63-addf-478cb67bc561\") " Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991551 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-data" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991568 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd23d352-acb4-4b63-addf-478cb67bc561-istiod-ca-cert" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.991680 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-credential-socket" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.992174 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd23d352-acb4-4b63-addf-478cb67bc561-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.992198 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.992213 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.992226 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:34.993192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.992238 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:34.993656 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.993633 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-envoy" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:34.993710 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.993641 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-istio-token" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:34.993744 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.993698 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-kube-api-access-r6zz7" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "kube-api-access-r6zz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:34.993872 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:34.993857 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd23d352-acb4-4b63-addf-478cb67bc561-istio-podinfo" pod "dd23d352-acb4-4b63-addf-478cb67bc561" (UID: "dd23d352-acb4-4b63-addf-478cb67bc561"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:01:35.093661 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.093623 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:35.093661 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.093651 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd23d352-acb4-4b63-addf-478cb67bc561-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:35.093661 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.093665 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd23d352-acb4-4b63-addf-478cb67bc561-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:35.093896 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.093680 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zz7\" (UniqueName: \"kubernetes.io/projected/dd23d352-acb4-4b63-addf-478cb67bc561-kube-api-access-r6zz7\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:35.366197 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.366162 6481 generic.go:356] "Generic (PLEG): container finished" podID="dd23d352-acb4-4b63-addf-478cb67bc561" containerID="abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59" exitCode=0 Oct 31 20:01:35.366395 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.366222 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" Oct 31 20:01:35.366395 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.366237 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" event={"ID":"dd23d352-acb4-4b63-addf-478cb67bc561","Type":"ContainerDied","Data":"abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59"} Oct 31 20:01:35.366395 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.366287 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f" event={"ID":"dd23d352-acb4-4b63-addf-478cb67bc561","Type":"ContainerDied","Data":"1307a8fcb6e8837c5873bd635671ef3194c1d5432143d62530996d416bf85b19"} Oct 31 20:01:35.366395 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.366308 6481 scope.go:122] "RemoveContainer" containerID="abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59" Oct 31 20:01:35.377574 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.377554 6481 scope.go:122] "RemoveContainer" containerID="9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888" Oct 31 20:01:35.386583 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.386566 6481 scope.go:122] "RemoveContainer" containerID="abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59" Oct 31 20:01:35.386872 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:35.386851 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59\": container with ID starting with abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59 not found: ID does not exist" containerID="abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59" Oct 31 20:01:35.386950 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.386885 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59"} err="failed to get container status \"abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59\": rpc error: code = NotFound desc = could not find container \"abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59\": container with ID starting with abcedb30c5ae1b8f2871630123a34fa847607c3522568f4dbff70710de3b4a59 not found: ID does not exist" Oct 31 20:01:35.386950 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.386910 6481 scope.go:122] "RemoveContainer" containerID="9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888" Oct 31 20:01:35.387231 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:35.387215 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888\": container with ID starting with 9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888 not found: ID does not exist" containerID="9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888" Oct 31 20:01:35.387288 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.387239 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888"} err="failed to get container status \"9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888\": rpc error: code = NotFound desc = could not find container \"9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888\": container with ID starting with 9165227af905a61378c5714a9b89d56e75d48860f8ce8c418357e823d3eee888 not found: ID does not exist" Oct 31 20:01:35.519708 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.519681 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f"] Oct 31 20:01:35.566063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:35.566032 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a5ba62-openshift-default-cc54b6d69-zsv7f"] Oct 31 20:01:36.887328 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:36.887281 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" path="/var/lib/kubelet/pods/dd23d352-acb4-4b63-addf-478cb67bc561/volumes" Oct 31 20:01:42.876255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:42.876214 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx"] Oct 31 20:01:42.886597 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:42.886552 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" podUID="fb10361c-e8f2-4c07-a044-0d8340acb152" containerName="maas-api" containerID="cri-o://e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7" gracePeriod=30 Oct 31 20:01:43.323593 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.323564 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 20:01:43.473017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.472416 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb10361c-e8f2-4c07-a044-0d8340acb152-kube-api-access-cvgjk\" (UniqueName: \"kubernetes.io/projected/fb10361c-e8f2-4c07-a044-0d8340acb152-kube-api-access-cvgjk\") pod \"fb10361c-e8f2-4c07-a044-0d8340acb152\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " Oct 31 20:01:43.473017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.472695 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb10361c-e8f2-4c07-a044-0d8340acb152-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10361c-e8f2-4c07-a044-0d8340acb152-openshift-service-ca\") pod \"fb10361c-e8f2-4c07-a044-0d8340acb152\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " Oct 31 20:01:43.473017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.472756 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls\") pod \"fb10361c-e8f2-4c07-a044-0d8340acb152\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " Oct 31 20:01:43.473017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.472852 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-metrics-certs\") pod \"fb10361c-e8f2-4c07-a044-0d8340acb152\" (UID: \"fb10361c-e8f2-4c07-a044-0d8340acb152\") " Oct 31 20:01:43.498457 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.498420 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb10361c-e8f2-4c07-a044-0d8340acb152-openshift-service-ca" pod "fb10361c-e8f2-4c07-a044-0d8340acb152" (UID: "fb10361c-e8f2-4c07-a044-0d8340acb152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:43.509833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.509580 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-metrics-certs" pod "fb10361c-e8f2-4c07-a044-0d8340acb152" (UID: "fb10361c-e8f2-4c07-a044-0d8340acb152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:43.510968 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.510940 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb10361c-e8f2-4c07-a044-0d8340acb152-kube-api-access-cvgjk" pod "fb10361c-e8f2-4c07-a044-0d8340acb152" (UID: "fb10361c-e8f2-4c07-a044-0d8340acb152"). InnerVolumeSpecName "kube-api-access-cvgjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:43.511863 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.511818 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls" pod "fb10361c-e8f2-4c07-a044-0d8340acb152" (UID: "fb10361c-e8f2-4c07-a044-0d8340acb152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:43.555261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.555226 6481 generic.go:356] "Generic (PLEG): container finished" podID="fb10361c-e8f2-4c07-a044-0d8340acb152" containerID="e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7" exitCode=0 Oct 31 20:01:43.555455 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.555393 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" event={"ID":"fb10361c-e8f2-4c07-a044-0d8340acb152","Type":"ContainerDied","Data":"e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7"} Oct 31 20:01:43.555455 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.555435 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" event={"ID":"fb10361c-e8f2-4c07-a044-0d8340acb152","Type":"ContainerDied","Data":"616cbd2a3e51856bd0481c927be30024eb653b38be3f46f2ac77719bc16bd6e5"} Oct 31 20:01:43.555585 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.555458 6481 scope.go:122] "RemoveContainer" containerID="e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7" Oct 31 20:01:43.555645 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.555590 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx" Oct 31 20:01:43.570636 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.570618 6481 scope.go:122] "RemoveContainer" containerID="e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7" Oct 31 20:01:43.570960 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:43.570937 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7\": container with ID starting with e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7 not found: ID does not exist" containerID="e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7" Oct 31 20:01:43.571077 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.570968 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7"} err="failed to get container status \"e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7\": rpc error: code = NotFound desc = could not find container \"e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7\": container with ID starting with e345750cbdec5e08d750c6a30aafc74d91ec662f667aa26ce2c3302d963916c7 not found: ID does not exist" Oct 31 20:01:43.574939 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.574918 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvgjk\" (UniqueName: \"kubernetes.io/projected/fb10361c-e8f2-4c07-a044-0d8340acb152-kube-api-access-cvgjk\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:43.574939 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.574939 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10361c-e8f2-4c07-a044-0d8340acb152-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:43.575074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.574950 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:43.575074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.574958 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10361c-e8f2-4c07-a044-0d8340acb152-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:43.713228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.713195 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx"] Oct 31 20:01:43.718382 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:43.718353 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1de410-56d87fd5cb-b72kx"] Oct 31 20:01:44.881215 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:44.881181 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb10361c-e8f2-4c07-a044-0d8340acb152" path="/var/lib/kubelet/pods/fb10361c-e8f2-4c07-a044-0d8340acb152/volumes" Oct 31 20:01:52.261436 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:52.261405 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw"] Oct 31 20:01:52.261922 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:52.261610 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" containerID="cri-o://efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885" gracePeriod=30 Oct 31 20:01:53.798384 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:53.798350 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:01:53.798801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:53.798411 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:01:57.325756 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:57.325718 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:01:57.555250 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.555222 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 20:01:57.648845 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.648813 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-socket\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.648867 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-data\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.648896 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istiod-ca-cert\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.648922 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-envoy\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.648973 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-token\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-token\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649042 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-certs\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649098 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-credential-socket\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649103 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-socket" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.649274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649209 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-data" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.649274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649254 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-kube-api-access-rqk97\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-kube-api-access-rqk97\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649322 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-podinfo\") pod \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\" (UID: \"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520\") " Oct 31 20:01:57.649529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649370 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istiod-ca-cert" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:57.649529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649398 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-credential-socket" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.649529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649409 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-certs" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.649899 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649873 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.649979 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649904 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.649979 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649918 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.649979 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649931 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.649979 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.649946 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.651410 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.651383 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-kube-api-access-rqk97" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "kube-api-access-rqk97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:57.651410 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.651401 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-token" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:57.651649 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.651630 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-envoy" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.651712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.651657 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-podinfo" pod "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" (UID: "dc1bd6b3-6ecf-426d-9b54-7a9fc4135520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:01:57.699957 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.699929 6481 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerID="efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885" exitCode=0 Oct 31 20:01:57.700099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.699988 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" Oct 31 20:01:57.700099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.700022 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" event={"ID":"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520","Type":"ContainerDied","Data":"efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885"} Oct 31 20:01:57.700186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.700120 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw" event={"ID":"dc1bd6b3-6ecf-426d-9b54-7a9fc4135520","Type":"ContainerDied","Data":"568f34da314b81bb1f06246c96ff7a5eb52a1e82cec6b8a0fc307649ea3c1aa3"} Oct 31 20:01:57.700186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.700149 6481 scope.go:122] "RemoveContainer" containerID="efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885" Oct 31 20:01:57.711275 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.711256 6481 scope.go:122] "RemoveContainer" containerID="87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32" Oct 31 20:01:57.720537 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.720497 6481 scope.go:122] "RemoveContainer" containerID="efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885" Oct 31 20:01:57.720774 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:57.720753 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885\": container with ID starting with efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885 not found: ID does not exist" containerID="efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885" Oct 31 20:01:57.720835 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.720782 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885"} err="failed to get container status \"efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885\": rpc error: code = NotFound desc = could not find container \"efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885\": container with ID starting with efd3ad588cabe77da606399ea2f2f42c3775d27840256889a9cd3581c2803885 not found: ID does not exist" Oct 31 20:01:57.720835 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.720799 6481 scope.go:122] "RemoveContainer" containerID="87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32" Oct 31 20:01:57.721037 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:01:57.721021 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32\": container with ID starting with 87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32 not found: ID does not exist" containerID="87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32" Oct 31 20:01:57.721101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.721040 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32"} err="failed to get container status \"87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32\": rpc error: code = NotFound desc = could not find container \"87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32\": container with ID starting with 87a0fd76b19e59b200bf966c9e2bdff3b221dd4f1b5fbb7de69cb115cfd40d32 not found: ID does not exist" Oct 31 20:01:57.751658 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.751632 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.751658 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.751657 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqk97\" (UniqueName: \"kubernetes.io/projected/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-kube-api-access-rqk97\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.751879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.751669 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.751879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.751677 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.828409 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw"] Oct 31 20:01:57.832701 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:57.832677 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1de410-openshift-default-7754f8d699-zs8vw"] Oct 31 20:01:58.880629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:01:58.880592 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" path="/var/lib/kubelet/pods/dc1bd6b3-6ecf-426d-9b54-7a9fc4135520/volumes" Oct 31 20:02:05.689284 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:05.689244 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg"] Oct 31 20:02:05.689756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:05.689444 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" podUID="05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" containerName="maas-api" containerID="cri-o://b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b" gracePeriod=30 Oct 31 20:02:05.971932 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:05.971549 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 20:02:06.006811 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.005705 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-openshift-service-ca\") pod \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " Oct 31 20:02:06.006811 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.005795 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls\") pod \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " Oct 31 20:02:06.006811 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.005923 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-metrics-certs\") pod \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " Oct 31 20:02:06.006811 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.006013 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-kube-api-access-rh9xz\" (UniqueName: \"kubernetes.io/projected/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-kube-api-access-rh9xz\") pod \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\" (UID: \"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd\") " Oct 31 20:02:06.010125 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.010072 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-openshift-service-ca" pod "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" (UID: "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:06.014796 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.014747 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-kube-api-access-rh9xz" pod "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" (UID: "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd"). InnerVolumeSpecName "kube-api-access-rh9xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:06.017062 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.017015 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-metrics-certs" pod "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" (UID: "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:06.018328 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.018305 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls" pod "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" (UID: "05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:06.108705 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.108670 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:06.108866 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.108709 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh9xz\" (UniqueName: \"kubernetes.io/projected/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-kube-api-access-rh9xz\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:06.108866 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.108727 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:06.108866 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.108742 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:06.765213 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.765177 6481 generic.go:356] "Generic (PLEG): container finished" podID="05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" containerID="b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b" exitCode=0 Oct 31 20:02:06.765636 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.765230 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" Oct 31 20:02:06.765636 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.765229 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" event={"ID":"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd","Type":"ContainerDied","Data":"b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b"} Oct 31 20:02:06.765636 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.765350 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg" event={"ID":"05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd","Type":"ContainerDied","Data":"68f0aaf6671e2c79e39216ba3d85723ced2086cf64d8e9f818626a8bc3683ab9"} Oct 31 20:02:06.765636 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.765379 6481 scope.go:122] "RemoveContainer" containerID="b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b" Oct 31 20:02:06.776921 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.776894 6481 scope.go:122] "RemoveContainer" containerID="b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b" Oct 31 20:02:06.777253 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:06.777234 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b\": container with ID starting with b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b not found: ID does not exist" containerID="b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b" Oct 31 20:02:06.777329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.777262 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b"} err="failed to get container status \"b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b\": rpc error: code = NotFound desc = could not find container \"b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b\": container with ID starting with b30782f488d8f63f3098617ea3a53e0b45099d5cffb63d64664bde9416ce267b not found: ID does not exist" Oct 31 20:02:06.957013 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.956218 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg"] Oct 31 20:02:06.990814 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:06.990779 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-fpnfg"] Oct 31 20:02:07.094489 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.094402 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn"] Oct 31 20:02:07.094657 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.094596 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" podUID="6103cd8a-4d26-40d7-b08c-e76492bd3d15" containerName="maas-api" containerID="cri-o://1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3" gracePeriod=30 Oct 31 20:02:07.428019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.427969 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 20:02:07.587016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.584691 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6103cd8a-4d26-40d7-b08c-e76492bd3d15-kube-api-access-42k5q\" (UniqueName: \"kubernetes.io/projected/6103cd8a-4d26-40d7-b08c-e76492bd3d15-kube-api-access-42k5q\") pod \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " Oct 31 20:02:07.587016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.584769 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls\") pod \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " Oct 31 20:02:07.587016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.584842 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-metrics-certs\") pod \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " Oct 31 20:02:07.587016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.585024 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6103cd8a-4d26-40d7-b08c-e76492bd3d15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6103cd8a-4d26-40d7-b08c-e76492bd3d15-openshift-service-ca\") pod \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\" (UID: \"6103cd8a-4d26-40d7-b08c-e76492bd3d15\") " Oct 31 20:02:07.588718 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.588222 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6103cd8a-4d26-40d7-b08c-e76492bd3d15-openshift-service-ca" pod "6103cd8a-4d26-40d7-b08c-e76492bd3d15" (UID: "6103cd8a-4d26-40d7-b08c-e76492bd3d15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:07.593573 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.593521 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6103cd8a-4d26-40d7-b08c-e76492bd3d15-kube-api-access-42k5q" pod "6103cd8a-4d26-40d7-b08c-e76492bd3d15" (UID: "6103cd8a-4d26-40d7-b08c-e76492bd3d15"). InnerVolumeSpecName "kube-api-access-42k5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:07.594239 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.594202 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls" pod "6103cd8a-4d26-40d7-b08c-e76492bd3d15" (UID: "6103cd8a-4d26-40d7-b08c-e76492bd3d15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:07.598965 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.596574 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-metrics-certs" pod "6103cd8a-4d26-40d7-b08c-e76492bd3d15" (UID: "6103cd8a-4d26-40d7-b08c-e76492bd3d15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:07.690299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.690148 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:07.690299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.690189 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6103cd8a-4d26-40d7-b08c-e76492bd3d15-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:07.690299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.690205 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42k5q\" (UniqueName: \"kubernetes.io/projected/6103cd8a-4d26-40d7-b08c-e76492bd3d15-kube-api-access-42k5q\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:07.690299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.690220 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6103cd8a-4d26-40d7-b08c-e76492bd3d15-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:07.774715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.774665 6481 generic.go:356] "Generic (PLEG): container finished" podID="6103cd8a-4d26-40d7-b08c-e76492bd3d15" containerID="1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3" exitCode=0 Oct 31 20:02:07.775175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.774744 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" Oct 31 20:02:07.775175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.774749 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" event={"ID":"6103cd8a-4d26-40d7-b08c-e76492bd3d15","Type":"ContainerDied","Data":"1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3"} Oct 31 20:02:07.775175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.774813 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn" event={"ID":"6103cd8a-4d26-40d7-b08c-e76492bd3d15","Type":"ContainerDied","Data":"cac6b08c5d71a09145d46e274abcf464c53891be62a0b01d720e4b3be21d8231"} Oct 31 20:02:07.775175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.774844 6481 scope.go:122] "RemoveContainer" containerID="1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3" Oct 31 20:02:07.786764 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.786749 6481 scope.go:122] "RemoveContainer" containerID="1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3" Oct 31 20:02:07.787066 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:07.787043 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3\": container with ID starting with 1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3 not found: ID does not exist" containerID="1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3" Oct 31 20:02:07.787147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.787076 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3"} err="failed to get container status \"1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3\": rpc error: code = NotFound desc = could not find container \"1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3\": container with ID starting with 1af9efd71bcf8f35fa8eeef82bad96f5f83c93a8846942458cb30025e5d200f3 not found: ID does not exist" Oct 31 20:02:07.925894 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.925862 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn"] Oct 31 20:02:07.971509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:07.971485 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-9a8346-668855f948-222rn"] Oct 31 20:02:08.880103 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:08.880063 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" path="/var/lib/kubelet/pods/05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd/volumes" Oct 31 20:02:08.880684 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:08.880664 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6103cd8a-4d26-40d7-b08c-e76492bd3d15" path="/var/lib/kubelet/pods/6103cd8a-4d26-40d7-b08c-e76492bd3d15/volumes" Oct 31 20:02:16.362671 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:16.362629 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk"] Oct 31 20:02:16.363126 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:16.362898 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" containerID="cri-o://80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0" gracePeriod=30 Oct 31 20:02:17.926558 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:17.926527 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:02:17.926968 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:17.926588 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:02:21.654446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.654419 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 20:02:21.725262 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725229 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-kube-api-access-fwrsb\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-kube-api-access-fwrsb\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725527 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725321 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-token\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-token\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725527 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725423 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-credential-socket\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725527 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725457 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-certs\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725592 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-podinfo\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725654 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istiod-ca-cert\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725721 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-socket\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725748 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-envoy\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.725808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.725776 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-data\") pod \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\" (UID: \"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e\") " Oct 31 20:02:21.726933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.726630 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istiod-ca-cert" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:21.726933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.726756 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-data" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:21.726933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.726805 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-socket" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:21.726933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.726855 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-certs" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:21.727287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.727119 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-credential-socket" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:21.728262 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.728230 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-kube-api-access-fwrsb" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "kube-api-access-fwrsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:21.728702 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.728681 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-podinfo" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:02:21.729220 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.729177 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-token" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:21.729608 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.729583 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-envoy" pod "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" (UID: "65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:21.828196 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828161 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828196 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828195 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828210 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828225 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828238 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828252 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828264 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828276 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.828476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.828289 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrsb\" (UniqueName: \"kubernetes.io/projected/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e-kube-api-access-fwrsb\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:21.977750 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.976233 6481 generic.go:356] "Generic (PLEG): container finished" podID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerID="80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0" exitCode=0 Oct 31 20:02:21.977750 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.976458 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" event={"ID":"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e","Type":"ContainerDied","Data":"80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0"} Oct 31 20:02:21.977750 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.976514 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" event={"ID":"65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e","Type":"ContainerDied","Data":"5b4e83fcf367f380f7c9f36fb9405598c92740043455809d90864a30966057a8"} Oct 31 20:02:21.977750 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.976543 6481 scope.go:122] "RemoveContainer" containerID="80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0" Oct 31 20:02:21.977750 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:21.976710 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk" Oct 31 20:02:22.015590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.015568 6481 scope.go:122] "RemoveContainer" containerID="0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb" Oct 31 20:02:22.024898 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.024871 6481 scope.go:122] "RemoveContainer" containerID="80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0" Oct 31 20:02:22.025222 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:22.025195 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0\": container with ID starting with 80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0 not found: ID does not exist" containerID="80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0" Oct 31 20:02:22.025289 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.025232 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0"} err="failed to get container status \"80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0\": rpc error: code = NotFound desc = could not find container \"80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0\": container with ID starting with 80fc56d77429076d5eea65abff29856b1b8a01954eea794a35dea430b57a2cf0 not found: ID does not exist" Oct 31 20:02:22.025289 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.025249 6481 scope.go:122] "RemoveContainer" containerID="0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb" Oct 31 20:02:22.025502 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:22.025485 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb\": container with ID starting with 0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb not found: ID does not exist" containerID="0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb" Oct 31 20:02:22.025549 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.025507 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb"} err="failed to get container status \"0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb\": rpc error: code = NotFound desc = could not find container \"0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb\": container with ID starting with 0c047202cdcc656a398ce715d683265e47fe0fd50a0b5bb6e8fcacc1668682eb not found: ID does not exist" Oct 31 20:02:22.162013 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.158091 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk"] Oct 31 20:02:22.165080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.164540 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9a8346-openshift-default-665c55db75-5pjbk"] Oct 31 20:02:22.241431 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.241349 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm"] Oct 31 20:02:22.246162 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.246135 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4"] Oct 31 20:02:22.880511 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.880482 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" path="/var/lib/kubelet/pods/65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e/volumes" Oct 31 20:02:22.932909 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.932876 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4"] Oct 31 20:02:22.933126 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:22.933083 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" containerID="cri-o://01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad" gracePeriod=30 Oct 31 20:02:24.891520 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:24.891489 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght"] Oct 31 20:02:24.891918 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:24.891678 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" containerID="cri-o://cc74997edef1f50b3044ee3192adde23f71e0ceb01777e28877bef8e927eaec0" gracePeriod=30 Oct 31 20:02:26.333349 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:26.333314 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn"] Oct 31 20:02:26.333769 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:26.333536 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" containerID="cri-o://5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8" gracePeriod=30 Oct 31 20:02:26.694104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:26.694021 6481 patch_prober.go:29] interesting pod/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:02:26.694104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:26.694086 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:02:27.325781 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:27.325747 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:02:28.206069 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.205987 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 20:02:28.325381 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.325353 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-kube-api-access-mq4cn\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-kube-api-access-mq4cn\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.325582 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.325387 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-podinfo\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.325582 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.325412 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-token\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-token\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.325582 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.325541 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-socket\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.325763 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.325701 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-credential-socket\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.326457 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326289 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-data\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.326457 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326389 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istiod-ca-cert\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.326627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326458 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-certs\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.326627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326517 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-data" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:28.326627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326536 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-credential-socket" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:28.326805 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326632 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-envoy\") pod \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\" (UID: \"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25\") " Oct 31 20:02:28.326805 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.325804 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-socket" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:28.326805 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326788 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-certs" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:28.326949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.326876 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istiod-ca-cert" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:28.329004 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.327882 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.329004 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.327922 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.329004 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.327938 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.329004 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.327958 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.329004 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.327971 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.330631 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.329423 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-token" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:28.337671 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.337641 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-envoy" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:28.338158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.338137 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-kube-api-access-mq4cn" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "kube-api-access-mq4cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:28.338619 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.338603 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-podinfo" pod "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" (UID: "3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:02:28.429508 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.429474 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.429508 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.429504 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq4cn\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-kube-api-access-mq4cn\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.429508 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.429515 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:28.429729 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:28.429523 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:29.032198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.032160 6481 generic.go:356] "Generic (PLEG): container finished" podID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerID="01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad" exitCode=0 Oct 31 20:02:29.032401 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.032228 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" Oct 31 20:02:29.032401 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.032240 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" event={"ID":"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25","Type":"ContainerDied","Data":"01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad"} Oct 31 20:02:29.032401 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.032307 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4" event={"ID":"3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25","Type":"ContainerDied","Data":"c7eee9d8bffcbf707d0d9477834bfff87d024a5a851ce1a8fde75e9a8bdcbbb0"} Oct 31 20:02:29.032401 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.032340 6481 scope.go:122] "RemoveContainer" containerID="01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad" Oct 31 20:02:29.043353 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.043335 6481 scope.go:122] "RemoveContainer" containerID="01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad" Oct 31 20:02:29.043606 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:29.043587 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad\": container with ID starting with 01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad not found: ID does not exist" containerID="01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad" Oct 31 20:02:29.043708 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.043617 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad"} err="failed to get container status \"01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad\": rpc error: code = NotFound desc = could not find container \"01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad\": container with ID starting with 01cc402ede2f0a4f031bf863958143ba79e4044eb34251e03f35c167041901ad not found: ID does not exist" Oct 31 20:02:29.185818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.185737 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4"] Oct 31 20:02:29.231663 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:29.231626 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8q6r4"] Oct 31 20:02:30.042942 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.042909 6481 generic.go:356] "Generic (PLEG): container finished" podID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerID="cc74997edef1f50b3044ee3192adde23f71e0ceb01777e28877bef8e927eaec0" exitCode=0 Oct 31 20:02:30.043422 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.042983 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" event={"ID":"41fed4fb-ffc4-42c8-a6b4-14c475306893","Type":"ContainerDied","Data":"cc74997edef1f50b3044ee3192adde23f71e0ceb01777e28877bef8e927eaec0"} Oct 31 20:02:30.161488 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.161464 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 20:02:30.252881 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.252848 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-certs\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.252934 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-envoy\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253046 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-data\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253133 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-certs" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253176 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-kube-api-access-9j244\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-kube-api-access-9j244\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253228 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-token\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-token\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253300 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41fed4fb-ffc4-42c8-a6b4-14c475306893-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fed4fb-ffc4-42c8-a6b4-14c475306893-istiod-ca-cert\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253312 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-data" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:30.253373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253352 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-credential-socket\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253760 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253402 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-socket\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253760 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253487 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-podinfo\") pod \"41fed4fb-ffc4-42c8-a6b4-14c475306893\" (UID: \"41fed4fb-ffc4-42c8-a6b4-14c475306893\") " Oct 31 20:02:30.253760 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253592 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-credential-socket" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:30.253760 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253661 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-socket" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:30.253760 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.253717 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fed4fb-ffc4-42c8-a6b4-14c475306893-istiod-ca-cert" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:30.254099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.254080 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fed4fb-ffc4-42c8-a6b4-14c475306893-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.254198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.254102 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.254198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.254116 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.254198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.254128 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.254198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.254141 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.255615 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.255589 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-envoy" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:30.255716 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.255592 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-kube-api-access-9j244" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "kube-api-access-9j244". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:30.255935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.255914 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-token" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:30.256043 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.255966 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-podinfo" pod "41fed4fb-ffc4-42c8-a6b4-14c475306893" (UID: "41fed4fb-ffc4-42c8-a6b4-14c475306893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:02:30.355142 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.355106 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.355142 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.355138 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.355142 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.355151 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j244\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-kube-api-access-9j244\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.355377 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.355164 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fed4fb-ffc4-42c8-a6b4-14c475306893-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:30.887505 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:30.887474 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" path="/var/lib/kubelet/pods/3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25/volumes" Oct 31 20:02:31.059494 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.059457 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" event={"ID":"41fed4fb-ffc4-42c8-a6b4-14c475306893","Type":"ContainerDied","Data":"aacff49a0dad3ae9af41c28c3089c87be8b6cfdb39543006ff888e030bc5dcff"} Oct 31 20:02:31.059796 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.059772 6481 scope.go:122] "RemoveContainer" containerID="cc74997edef1f50b3044ee3192adde23f71e0ceb01777e28877bef8e927eaec0" Oct 31 20:02:31.060201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.060173 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght" Oct 31 20:02:31.267604 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.267568 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght"] Oct 31 20:02:31.274346 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.274321 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cgght"] Oct 31 20:02:31.604218 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.604195 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 20:02:31.674367 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674323 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-certs\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.674367 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674370 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-credential-socket\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.674635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674406 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-socket\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.674635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674442 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-kube-api-access-q47b6\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-kube-api-access-q47b6\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.674635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674543 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-envoy\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.674635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674605 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-podinfo\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.674879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674623 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-certs" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:31.674879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674717 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-socket" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:31.674879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674762 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-credential-socket" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:31.674879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674767 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istiod-ca-cert\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.675164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674934 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-token\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-token\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.675164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.674971 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-data\") pod \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\" (UID: \"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775\") " Oct 31 20:02:31.675164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.675127 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istiod-ca-cert" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:31.675330 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.675265 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-data" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:31.675703 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.675676 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.675788 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.675707 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.675788 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.675723 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.675788 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.675736 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.675788 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.675750 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.676970 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.676943 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-kube-api-access-q47b6" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "kube-api-access-q47b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:31.677124 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.676985 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-envoy" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:31.677124 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.677028 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-podinfo" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:02:31.677124 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.677052 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-token" pod "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" (UID: "cd7850a1-eb98-4f34-b9c3-b0eae9bd8775"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:31.777092 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.777060 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47b6\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-kube-api-access-q47b6\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.777092 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.777088 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.777092 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.777098 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.777339 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.777106 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:31.845762 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.845682 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl"] Oct 31 20:02:31.846276 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846261 6481 cpu_manager.go:414] "containerMap: removing container" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846278 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846293 6481 cpu_manager.go:414] "containerMap: removing container" podUID="56c22de4-a8ca-42ea-991b-857382cba34a" containerName="payload-processing" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846298 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c22de4-a8ca-42ea-991b-857382cba34a" containerName="payload-processing" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846305 6481 cpu_manager.go:414] "containerMap: removing container" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846309 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846316 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ca110c79-031f-4725-8471-6dc641713674" containerName="payload-pre-processing" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846321 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca110c79-031f-4725-8471-6dc641713674" containerName="payload-pre-processing" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846329 6481 cpu_manager.go:414] "containerMap: removing container" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" Oct 31 20:02:31.846329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846333 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846343 6481 cpu_manager.go:414] "containerMap: removing container" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846348 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846356 6481 cpu_manager.go:414] "containerMap: removing container" podUID="05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846360 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846368 6481 cpu_manager.go:414] "containerMap: removing container" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846372 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846384 6481 cpu_manager.go:414] "containerMap: removing container" podUID="fb10361c-e8f2-4c07-a044-0d8340acb152" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846388 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb10361c-e8f2-4c07-a044-0d8340acb152" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846397 6481 cpu_manager.go:414] "containerMap: removing container" podUID="6103cd8a-4d26-40d7-b08c-e76492bd3d15" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846402 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6103cd8a-4d26-40d7-b08c-e76492bd3d15" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846412 6481 cpu_manager.go:414] "containerMap: removing container" podUID="798bcb87-4cb1-4aac-8e6c-04033dff6888" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846416 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798bcb87-4cb1-4aac-8e6c-04033dff6888" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846425 6481 cpu_manager.go:414] "containerMap: removing container" podUID="7532c752-6a84-43a7-ba46-b6c38c03f8c7" containerName="payload-processing" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846429 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7532c752-6a84-43a7-ba46-b6c38c03f8c7" containerName="payload-processing" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846435 6481 cpu_manager.go:414] "containerMap: removing container" podUID="6d113153-5532-437d-b16b-c30fa0c8721f" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846439 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d113153-5532-437d-b16b-c30fa0c8721f" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846452 6481 cpu_manager.go:414] "containerMap: removing container" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846456 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846466 6481 cpu_manager.go:414] "containerMap: removing container" podUID="37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" containerName="payload-pre-processing" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846471 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" containerName="payload-pre-processing" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846482 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846486 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846492 6481 cpu_manager.go:414] "containerMap: removing container" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" containerName="test-curl-11074-b5cc64" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846496 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" containerName="test-curl-11074-b5cc64" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846577 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="798bcb87-4cb1-4aac-8e6c-04033dff6888" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846588 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7532c752-6a84-43a7-ba46-b6c38c03f8c7" containerName="payload-processing" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846594 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d113153-5532-437d-b16b-c30fa0c8721f" containerName="maas-api" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846602 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerName="istio-proxy" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846608 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b05dd8-cfb2-4dc5-9eb1-17a09058d4bf" containerName="payload-pre-processing" Oct 31 20:02:31.846601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846616 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846621 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfe4b38-38bc-4278-b1b2-3cf3e41b8c25" containerName="istio-proxy" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846627 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec00e11-5f4e-47a7-880d-2703d72a6892" containerName="test-curl-11074-b5cc64" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846633 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846640 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c22de4-a8ca-42ea-991b-857382cba34a" containerName="payload-processing" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846646 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846660 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca110c79-031f-4725-8471-6dc641713674" containerName="payload-pre-processing" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846666 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846673 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1bd6b3-6ecf-426d-9b54-7a9fc4135520" containerName="istio-proxy" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846678 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="05edfa8f-5c0d-4cf5-8e01-3e36de09a0cd" containerName="maas-api" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846685 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" containerName="istio-proxy" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846709 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb10361c-e8f2-4c07-a044-0d8340acb152" containerName="maas-api" Oct 31 20:02:31.847590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.846715 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6103cd8a-4d26-40d7-b08c-e76492bd3d15" containerName="maas-api" Oct 31 20:02:31.850403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.850382 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.901126 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.901091 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fd95d7-openshift-default-dockercfg-cvnc8\"" Oct 31 20:02:31.904050 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.904021 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl"] Oct 31 20:02:31.979393 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979355 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm5bl\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-kube-api-access-hm5bl\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.979393 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979406 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.979662 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979503 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.979662 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979545 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.979662 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979634 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.979827 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979762 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.979885 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979821 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.979885 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979852 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:31.980015 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:31.979888 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.068083 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.068047 6481 generic.go:356] "Generic (PLEG): container finished" podID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" containerID="5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8" exitCode=0 Oct 31 20:02:32.068256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.068115 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" Oct 31 20:02:32.068256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.068135 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" event={"ID":"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775","Type":"ContainerDied","Data":"5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8"} Oct 31 20:02:32.068256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.068212 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn" event={"ID":"cd7850a1-eb98-4f34-b9c3-b0eae9bd8775","Type":"ContainerDied","Data":"618d2c31c2e79f1e27e13167df381ba51ade4ce33e5d19bbd5251c2b2e48ce5d"} Oct 31 20:02:32.068256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.068239 6481 scope.go:122] "RemoveContainer" containerID="5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8" Oct 31 20:02:32.079333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.079315 6481 scope.go:122] "RemoveContainer" containerID="5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8" Oct 31 20:02:32.079623 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:32.079601 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8\": container with ID starting with 5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8 not found: ID does not exist" containerID="5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8" Oct 31 20:02:32.079668 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.079632 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8"} err="failed to get container status \"5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8\": rpc error: code = NotFound desc = could not find container \"5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8\": container with ID starting with 5f3def5aefc8c14df84661e6da7100ff8dc806fb3392fcb75f1ecb4bd73cbef8 not found: ID does not exist" Oct 31 20:02:32.081277 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081253 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081384 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081291 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081465 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081448 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm5bl\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-kube-api-access-hm5bl\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081512 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081495 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081552 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081619 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081591 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081679 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081637 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081710 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-data\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081779 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081723 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081828 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081777 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.081917 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081893 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.082042 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.081981 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-certs\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.082091 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.082057 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istiod-ca-cert\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.082130 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.082094 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-credential-socket\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.083772 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.083749 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-envoy\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.084005 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.083979 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-podinfo\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.134079 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.133988 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-token\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.134228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.134151 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm5bl\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-kube-api-access-hm5bl\") pod \"e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.164901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.164867 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:32.231494 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.231463 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn"] Oct 31 20:02:32.236167 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.236139 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-cmhjn"] Oct 31 20:02:32.581676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.581652 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl"] Oct 31 20:02:32.582878 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:02:32.582853 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6ca183_e38a_43ba_bd2e_8d444dbcc29a.slice/crio-6c65e6cfaa3b453a19f6d3e02dae3e71450981edb943594bcd31c96754499ed0 WatchSource:0}: Error finding container 6c65e6cfaa3b453a19f6d3e02dae3e71450981edb943594bcd31c96754499ed0: Status 404 returned error can't find the container with id 6c65e6cfaa3b453a19f6d3e02dae3e71450981edb943594bcd31c96754499ed0 Oct 31 20:02:32.585587 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.585556 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:02:32.585641 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.585625 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:02:32.585692 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.585652 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:02:32.882961 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.882882 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41fed4fb-ffc4-42c8-a6b4-14c475306893" path="/var/lib/kubelet/pods/41fed4fb-ffc4-42c8-a6b4-14c475306893/volumes" Oct 31 20:02:32.883509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:32.883493 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7850a1-eb98-4f34-b9c3-b0eae9bd8775" path="/var/lib/kubelet/pods/cd7850a1-eb98-4f34-b9c3-b0eae9bd8775/volumes" Oct 31 20:02:33.078046 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:33.078008 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" event={"ID":"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a","Type":"ContainerStarted","Data":"d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37"} Oct 31 20:02:33.078233 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:33.078062 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" event={"ID":"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a","Type":"ContainerStarted","Data":"6c65e6cfaa3b453a19f6d3e02dae3e71450981edb943594bcd31c96754499ed0"} Oct 31 20:02:33.165293 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:33.165192 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:33.166933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:33.166905 6481 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 20:02:33.167102 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:33.166961 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 20:02:33.187208 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:33.187164 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podStartSLOduration=2.187151916 podStartE2EDuration="2.187151916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:33.18628686 +0000 UTC m=+2724.426510025" watchObservedRunningTime="2026-10-31 20:02:33.187151916 +0000 UTC m=+2724.427375044" Oct 31 20:02:34.171019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:34.165381 6481 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 20:02:34.171019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:34.165459 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 20:02:35.165236 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:35.165202 6481 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 20:02:35.165501 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:35.165473 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 20:02:36.190174 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:36.190127 6481 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 20:02:36.190677 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:36.190226 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 20:02:37.189132 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.189091 6481 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 20:02:37.189417 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.189390 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 20:02:37.264293 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.264246 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" containerID="cri-o://db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b" gracePeriod=45 Oct 31 20:02:37.291330 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.291280 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" containerID="cri-o://cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45" gracePeriod=45 Oct 31 20:02:37.448662 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.448568 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Oct 31 20:02:37.608694 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.608672 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 20:02:37.632565 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.632535 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 20:02:37.704333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704243 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd61e308-a1a7-43df-963c-669ecabe7e77-kube-api-access-rtdff\" (UniqueName: \"kubernetes.io/projected/cd61e308-a1a7-43df-963c-669ecabe7e77-kube-api-access-rtdff\") pod \"cd61e308-a1a7-43df-963c-669ecabe7e77\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " Oct 31 20:02:37.704333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704307 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-model-cache\") pod \"69df7514-491c-45c4-bb68-2677a787f0da\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " Oct 31 20:02:37.704570 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704339 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd61e308-a1a7-43df-963c-669ecabe7e77-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61e308-a1a7-43df-963c-669ecabe7e77-tls-certs\") pod \"cd61e308-a1a7-43df-963c-669ecabe7e77\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " Oct 31 20:02:37.704570 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704376 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-dshm\") pod \"69df7514-491c-45c4-bb68-2677a787f0da\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " Oct 31 20:02:37.704570 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704411 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-tmp-dir\") pod \"cd61e308-a1a7-43df-963c-669ecabe7e77\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " Oct 31 20:02:37.704570 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704460 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-home\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-home\") pod \"69df7514-491c-45c4-bb68-2677a787f0da\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " Oct 31 20:02:37.704570 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704530 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-dshm\") pod \"cd61e308-a1a7-43df-963c-669ecabe7e77\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " Oct 31 20:02:37.704833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704571 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-home\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-home\") pod \"cd61e308-a1a7-43df-963c-669ecabe7e77\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " Oct 31 20:02:37.704833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704602 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-model-cache\") pod \"cd61e308-a1a7-43df-963c-669ecabe7e77\" (UID: \"cd61e308-a1a7-43df-963c-669ecabe7e77\") " Oct 31 20:02:37.704833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704596 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-model-cache" pod "69df7514-491c-45c4-bb68-2677a787f0da" (UID: "69df7514-491c-45c4-bb68-2677a787f0da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.704833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704709 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-tmp-dir\") pod \"69df7514-491c-45c4-bb68-2677a787f0da\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " Oct 31 20:02:37.704833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704768 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69df7514-491c-45c4-bb68-2677a787f0da-tls-certs\" (UniqueName: \"kubernetes.io/secret/69df7514-491c-45c4-bb68-2677a787f0da-tls-certs\") pod \"69df7514-491c-45c4-bb68-2677a787f0da\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " Oct 31 20:02:37.705097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704831 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-home" pod "69df7514-491c-45c4-bb68-2677a787f0da" (UID: "69df7514-491c-45c4-bb68-2677a787f0da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.705097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704927 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69df7514-491c-45c4-bb68-2677a787f0da-kube-api-access-rv46q\" (UniqueName: \"kubernetes.io/projected/69df7514-491c-45c4-bb68-2677a787f0da-kube-api-access-rv46q\") pod \"69df7514-491c-45c4-bb68-2677a787f0da\" (UID: \"69df7514-491c-45c4-bb68-2677a787f0da\") " Oct 31 20:02:37.705097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.704951 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-home" pod "cd61e308-a1a7-43df-963c-669ecabe7e77" (UID: "cd61e308-a1a7-43df-963c-669ecabe7e77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.705097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705013 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-tmp-dir" pod "cd61e308-a1a7-43df-963c-669ecabe7e77" (UID: "cd61e308-a1a7-43df-963c-669ecabe7e77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.705097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705073 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-model-cache" pod "cd61e308-a1a7-43df-963c-669ecabe7e77" (UID: "cd61e308-a1a7-43df-963c-669ecabe7e77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.705361 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705174 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-tmp-dir" pod "69df7514-491c-45c4-bb68-2677a787f0da" (UID: "69df7514-491c-45c4-bb68-2677a787f0da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.705508 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705491 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.705577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705515 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.705577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705529 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.705577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705541 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.705577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705554 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.705577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.705567 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.707246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.707218 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-dshm" pod "69df7514-491c-45c4-bb68-2677a787f0da" (UID: "69df7514-491c-45c4-bb68-2677a787f0da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.707416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.707396 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd61e308-a1a7-43df-963c-669ecabe7e77-kube-api-access-rtdff" pod "cd61e308-a1a7-43df-963c-669ecabe7e77" (UID: "cd61e308-a1a7-43df-963c-669ecabe7e77"). InnerVolumeSpecName "kube-api-access-rtdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:37.707492 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.707437 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69df7514-491c-45c4-bb68-2677a787f0da-tls-certs" pod "69df7514-491c-45c4-bb68-2677a787f0da" (UID: "69df7514-491c-45c4-bb68-2677a787f0da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:37.707629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.707611 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-dshm" pod "cd61e308-a1a7-43df-963c-669ecabe7e77" (UID: "cd61e308-a1a7-43df-963c-669ecabe7e77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:37.708578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.708559 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69df7514-491c-45c4-bb68-2677a787f0da-kube-api-access-rv46q" pod "69df7514-491c-45c4-bb68-2677a787f0da" (UID: "69df7514-491c-45c4-bb68-2677a787f0da"). InnerVolumeSpecName "kube-api-access-rv46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:37.708719 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.708699 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd61e308-a1a7-43df-963c-669ecabe7e77-tls-certs" pod "cd61e308-a1a7-43df-963c-669ecabe7e77" (UID: "cd61e308-a1a7-43df-963c-669ecabe7e77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:37.807075 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.806913 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv46q\" (UniqueName: \"kubernetes.io/projected/69df7514-491c-45c4-bb68-2677a787f0da-kube-api-access-rv46q\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.807075 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.806949 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdff\" (UniqueName: \"kubernetes.io/projected/cd61e308-a1a7-43df-963c-669ecabe7e77-kube-api-access-rtdff\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.807075 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.806967 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd61e308-a1a7-43df-963c-669ecabe7e77-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.807075 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.806981 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69df7514-491c-45c4-bb68-2677a787f0da-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.807075 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.807012 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd61e308-a1a7-43df-963c-669ecabe7e77-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:37.807075 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:37.807026 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69df7514-491c-45c4-bb68-2677a787f0da-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:38.125694 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.125661 6481 generic.go:356] "Generic (PLEG): container finished" podID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerID="cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45" exitCode=0 Oct 31 20:02:38.125887 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.125710 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" Oct 31 20:02:38.125887 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.125747 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" event={"ID":"cd61e308-a1a7-43df-963c-669ecabe7e77","Type":"ContainerDied","Data":"cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45"} Oct 31 20:02:38.125887 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.125790 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" event={"ID":"cd61e308-a1a7-43df-963c-669ecabe7e77","Type":"ContainerDied","Data":"f62d1c8ca98a1ad927ffc90400d05c356a63bfecd6172facbf9704135dde2c1b"} Oct 31 20:02:38.125887 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.125811 6481 scope.go:122] "RemoveContainer" containerID="cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45" Oct 31 20:02:38.127406 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.127384 6481 generic.go:356] "Generic (PLEG): container finished" podID="69df7514-491c-45c4-bb68-2677a787f0da" containerID="db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b" exitCode=0 Oct 31 20:02:38.127498 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.127470 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" event={"ID":"69df7514-491c-45c4-bb68-2677a787f0da","Type":"ContainerDied","Data":"db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b"} Oct 31 20:02:38.127555 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.127500 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" event={"ID":"69df7514-491c-45c4-bb68-2677a787f0da","Type":"ContainerDied","Data":"6360c42cb596c821bff421c63969fb6510859ce5fabc66955e5c3be79c6d9849"} Oct 31 20:02:38.127607 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.127562 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm" Oct 31 20:02:38.138415 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.138399 6481 scope.go:122] "RemoveContainer" containerID="cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45" Oct 31 20:02:38.138720 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:38.138682 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45\": container with ID starting with cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45 not found: ID does not exist" containerID="cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45" Oct 31 20:02:38.138791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.138733 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45"} err="failed to get container status \"cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45\": rpc error: code = NotFound desc = could not find container \"cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45\": container with ID starting with cf1e86746420d2b8cb4f5fd5574e1fa25dbbe6f3d3f8efdb5e09f5a897cd8a45 not found: ID does not exist" Oct 31 20:02:38.138791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.138757 6481 scope.go:122] "RemoveContainer" containerID="db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b" Oct 31 20:02:38.149726 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.149705 6481 scope.go:122] "RemoveContainer" containerID="db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b" Oct 31 20:02:38.150044 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:38.150023 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b\": container with ID starting with db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b not found: ID does not exist" containerID="db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b" Oct 31 20:02:38.150117 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.150054 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b"} err="failed to get container status \"db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b\": rpc error: code = NotFound desc = could not find container \"db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b\": container with ID starting with db0d2a41fad5197e3e9e3d4e1cdadec1837c27c5787f400e288546a2a4d56a3b not found: ID does not exist" Oct 31 20:02:38.194956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.191130 6481 patch_prober.go:29] interesting pod/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 20:02:38.194956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.191203 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 20:02:38.296074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.295977 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4"] Oct 31 20:02:38.299367 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.299339 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4"] Oct 31 20:02:38.400122 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.400038 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm"] Oct 31 20:02:38.404190 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.404164 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7cfafd/distinct-2-w4-7cfafd-kserve-7cc4bf6b67-bxcsm"] Oct 31 20:02:38.615104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.615063 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7cfafd/distinct-w4-7cfafd-kserve-688c8b8f44-ssgx4" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:02:38.880147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.880112 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69df7514-491c-45c4-bb68-2677a787f0da" path="/var/lib/kubelet/pods/69df7514-491c-45c4-bb68-2677a787f0da/volumes" Oct 31 20:02:38.880604 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:38.880589 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" path="/var/lib/kubelet/pods/cd61e308-a1a7-43df-963c-669ecabe7e77/volumes" Oct 31 20:02:39.171283 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:39.171204 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:39.171676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:39.171656 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:39.172397 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:39.172374 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:02:46.102678 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.102628 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp"] Oct 31 20:02:46.103382 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.102927 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" podUID="5bcd3a97-c608-434c-a37f-bfbc79806c2f" containerName="maas-api" containerID="cri-o://7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8" gracePeriod=30 Oct 31 20:02:46.427751 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.427726 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 20:02:46.524128 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.524092 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-metrics-certs\") pod \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " Oct 31 20:02:46.524128 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.524131 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bcd3a97-c608-434c-a37f-bfbc79806c2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd3a97-c608-434c-a37f-bfbc79806c2f-openshift-service-ca\") pod \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " Oct 31 20:02:46.524333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.524192 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls\") pod \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " Oct 31 20:02:46.524333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.524226 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bcd3a97-c608-434c-a37f-bfbc79806c2f-kube-api-access-bswr7\" (UniqueName: \"kubernetes.io/projected/5bcd3a97-c608-434c-a37f-bfbc79806c2f-kube-api-access-bswr7\") pod \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\" (UID: \"5bcd3a97-c608-434c-a37f-bfbc79806c2f\") " Oct 31 20:02:46.524603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.524568 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bcd3a97-c608-434c-a37f-bfbc79806c2f-openshift-service-ca" pod "5bcd3a97-c608-434c-a37f-bfbc79806c2f" (UID: "5bcd3a97-c608-434c-a37f-bfbc79806c2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:46.524804 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.524786 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd3a97-c608-434c-a37f-bfbc79806c2f-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:46.526186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.526163 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-metrics-certs" pod "5bcd3a97-c608-434c-a37f-bfbc79806c2f" (UID: "5bcd3a97-c608-434c-a37f-bfbc79806c2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:46.526730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.526687 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls" pod "5bcd3a97-c608-434c-a37f-bfbc79806c2f" (UID: "5bcd3a97-c608-434c-a37f-bfbc79806c2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:46.526811 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.526751 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcd3a97-c608-434c-a37f-bfbc79806c2f-kube-api-access-bswr7" pod "5bcd3a97-c608-434c-a37f-bfbc79806c2f" (UID: "5bcd3a97-c608-434c-a37f-bfbc79806c2f"). InnerVolumeSpecName "kube-api-access-bswr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:46.636645 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.634214 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:46.636645 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.634247 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcd3a97-c608-434c-a37f-bfbc79806c2f-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:46.636645 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:46.634265 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bswr7\" (UniqueName: \"kubernetes.io/projected/5bcd3a97-c608-434c-a37f-bfbc79806c2f-kube-api-access-bswr7\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:47.220014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.216049 6481 generic.go:356] "Generic (PLEG): container finished" podID="5bcd3a97-c608-434c-a37f-bfbc79806c2f" containerID="7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8" exitCode=0 Oct 31 20:02:47.220014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.216243 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" event={"ID":"5bcd3a97-c608-434c-a37f-bfbc79806c2f","Type":"ContainerDied","Data":"7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8"} Oct 31 20:02:47.220014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.216299 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" event={"ID":"5bcd3a97-c608-434c-a37f-bfbc79806c2f","Type":"ContainerDied","Data":"ed050a27d5e2cf3a0486c8ee75c06b1f2e96a7ae95c827c14a9a14369b896abc"} Oct 31 20:02:47.220014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.216328 6481 scope.go:122] "RemoveContainer" containerID="7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8" Oct 31 20:02:47.220014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.216491 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp" Oct 31 20:02:47.234017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.233633 6481 scope.go:122] "RemoveContainer" containerID="7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8" Oct 31 20:02:47.234418 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:47.234251 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8\": container with ID starting with 7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8 not found: ID does not exist" containerID="7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8" Oct 31 20:02:47.234418 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.234294 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8"} err="failed to get container status \"7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8\": rpc error: code = NotFound desc = could not find container \"7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8\": container with ID starting with 7f5b3a9580d53c7b9e655de735f2be504e0420af37f359bdb45b880db7970ba8 not found: ID does not exist" Oct 31 20:02:47.390505 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.390469 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp"] Oct 31 20:02:47.434624 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:47.434592 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ddb5a8-54d4ff5455-pz8cp"] Oct 31 20:02:48.880412 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:48.880378 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bcd3a97-c608-434c-a37f-bfbc79806c2f" path="/var/lib/kubelet/pods/5bcd3a97-c608-434c-a37f-bfbc79806c2f/volumes" Oct 31 20:02:48.991089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:48.991048 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7"] Oct 31 20:02:48.991315 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:48.991289 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" containerID="cri-o://ffe3c9e0f098d15d503900db1adf7910db78a16d41d7aa399a2b853f122441d0" gracePeriod=30 Oct 31 20:02:51.385437 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:51.385400 6481 patch_prober.go:29] interesting pod/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:02:51.385832 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:51.385454 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:02:54.262025 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.261959 6481 generic.go:356] "Generic (PLEG): container finished" podID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerID="ffe3c9e0f098d15d503900db1adf7910db78a16d41d7aa399a2b853f122441d0" exitCode=0 Oct 31 20:02:54.262436 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.262037 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" event={"ID":"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc","Type":"ContainerDied","Data":"ffe3c9e0f098d15d503900db1adf7910db78a16d41d7aa399a2b853f122441d0"} Oct 31 20:02:54.262436 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.262082 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" event={"ID":"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc","Type":"ContainerDied","Data":"ffe589b2789a0c0f5550af3bbb34b91fc9e94c5c2e19e364ade8110da148778c"} Oct 31 20:02:54.262436 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.262099 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe589b2789a0c0f5550af3bbb34b91fc9e94c5c2e19e364ade8110da148778c" Oct 31 20:02:54.262436 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.262112 6481 scope.go:122] "RemoveContainer" containerID="22c2f61f4dbd721813592284365975197739d928393ff61cccf10e3d3eb3c47d" Oct 31 20:02:54.280493 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.280470 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 20:02:54.338374 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338343 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-podinfo\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338384 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-socket\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338430 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-credential-socket\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338520 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istiod-ca-cert\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338544 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-data\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338580 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-certs\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338779 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338610 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-kube-api-access-nlck5\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-kube-api-access-nlck5\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338779 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338640 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-token\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-token\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338779 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338715 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-socket" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:54.338943 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338793 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-envoy\") pod \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\" (UID: \"dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc\") " Oct 31 20:02:54.338943 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338821 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-credential-socket" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:54.338943 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338869 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-certs" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:54.339141 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.338955 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istiod-ca-cert" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:54.339141 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.339066 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-data" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:54.339327 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.339308 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.339396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.339329 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.339396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.339343 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.339396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.339358 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.339396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.339370 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.341070 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.341037 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-kube-api-access-nlck5" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "kube-api-access-nlck5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:54.341193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.341173 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-envoy" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:54.341257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.341205 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-token" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:54.342102 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.342083 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-podinfo" pod "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" (UID: "dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:02:54.440281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.440243 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlck5\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-kube-api-access-nlck5\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.440281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.440271 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.440281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.440280 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:54.440281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:54.440288 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:55.269753 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:55.269723 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7" Oct 31 20:02:55.526450 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:55.526367 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7"] Oct 31 20:02:55.531573 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:55.531528 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dfe658-openshift-default-f7b8568d9-dwpm7"] Oct 31 20:02:55.630159 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:55.630124 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr"] Oct 31 20:02:55.630402 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:55.630372 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" containerID="cri-o://87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02" gracePeriod=30 Oct 31 20:02:56.880918 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:02:56.880882 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" path="/var/lib/kubelet/pods/dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc/volumes" Oct 31 20:02:57.326366 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:02:57.326332 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:03:00.898963 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.898894 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 20:03:00.971533 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.971503 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-podinfo\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.971715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.971568 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-socket\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.971776 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.971721 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-kube-api-access-fh9cm\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-kube-api-access-fh9cm\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.971776 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.971755 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istiod-ca-cert\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.971884 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.971775 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-envoy\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.971947 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.971907 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-socket" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:00.971947 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.971910 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-certs\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.972115 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972095 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-data\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.972186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972140 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-certs" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:00.972186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972172 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-token\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-token\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.972284 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972173 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istiod-ca-cert" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:00.972284 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972261 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-credential-socket\") pod \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\" (UID: \"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9\") " Oct 31 20:03:00.972420 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972400 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-data" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:00.972745 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972704 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-credential-socket" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:00.972990 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.972970 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:00.973109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.973023 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:00.973109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.973038 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:00.973109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.973051 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:00.973109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.973064 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:00.973857 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.973835 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-podinfo" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:03:00.974318 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.974288 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-kube-api-access-fh9cm" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "kube-api-access-fh9cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:00.974409 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.974383 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-token" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:00.975381 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:00.975361 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-envoy" pod "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" (UID: "fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:01.074308 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.074270 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:01.074308 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.074306 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:01.074498 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.074324 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh9cm\" (UniqueName: \"kubernetes.io/projected/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-kube-api-access-fh9cm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:01.074498 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.074340 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:01.349144 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.349103 6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerID="87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02" exitCode=0 Oct 31 20:03:01.349357 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.349171 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" Oct 31 20:03:01.349357 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.349190 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" event={"ID":"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9","Type":"ContainerDied","Data":"87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02"} Oct 31 20:03:01.349357 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.349286 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr" event={"ID":"fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9","Type":"ContainerDied","Data":"f0beaec6771d174ba4bf8c831aefb265ebbef75ebd0156e64c26b67c75cb4922"} Oct 31 20:03:01.349357 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.349318 6481 scope.go:122] "RemoveContainer" containerID="87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02" Oct 31 20:03:01.364187 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.364169 6481 scope.go:122] "RemoveContainer" containerID="eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93" Oct 31 20:03:01.373170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.373155 6481 scope.go:122] "RemoveContainer" containerID="87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02" Oct 31 20:03:01.373449 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:03:01.373429 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02\": container with ID starting with 87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02 not found: ID does not exist" containerID="87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02" Oct 31 20:03:01.373516 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.373458 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02"} err="failed to get container status \"87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02\": rpc error: code = NotFound desc = could not find container \"87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02\": container with ID starting with 87055ae70d8bcb4e1987bcfc401b65883183728c45d51a8e449620ae44d7ef02 not found: ID does not exist" Oct 31 20:03:01.373516 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.373475 6481 scope.go:122] "RemoveContainer" containerID="eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93" Oct 31 20:03:01.373730 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:03:01.373714 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93\": container with ID starting with eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93 not found: ID does not exist" containerID="eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93" Oct 31 20:03:01.373780 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.373735 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93"} err="failed to get container status \"eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93\": rpc error: code = NotFound desc = could not find container \"eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93\": container with ID starting with eacf8390070d73535c61f9c452edb70a40cea58ce76ffc491a3347dd2ea6ba93 not found: ID does not exist" Oct 31 20:03:01.502439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.502408 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr"] Oct 31 20:03:01.548462 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:01.548424 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ddb5a8-openshift-default-6b9bf6b997-n6sdr"] Oct 31 20:03:02.879769 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:02.879731 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" path="/var/lib/kubelet/pods/fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9/volumes" Oct 31 20:03:11.376386 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376352 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82"] Oct 31 20:03:11.376889 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376876 6481 cpu_manager.go:414] "containerMap: removing container" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" Oct 31 20:03:11.376935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376890 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" Oct 31 20:03:11.376935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376903 6481 cpu_manager.go:414] "containerMap: removing container" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" Oct 31 20:03:11.376935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376909 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" Oct 31 20:03:11.376935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376920 6481 cpu_manager.go:414] "containerMap: removing container" podUID="5bcd3a97-c608-434c-a37f-bfbc79806c2f" containerName="maas-api" Oct 31 20:03:11.376935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376925 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bcd3a97-c608-434c-a37f-bfbc79806c2f" containerName="maas-api" Oct 31 20:03:11.376935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376939 6481 cpu_manager.go:414] "containerMap: removing container" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376943 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376958 6481 cpu_manager.go:414] "containerMap: removing container" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376962 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ef4a4d-8d1c-4399-8d38-a8cfdea2ce6e" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376968 6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376972 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376978 6481 cpu_manager.go:414] "containerMap: removing container" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.376982 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377032 6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377037 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377110 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377118 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377124 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd61e308-a1a7-43df-963c-669ecabe7e77" containerName="main" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377132 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377139 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bcd3a97-c608-434c-a37f-bfbc79806c2f" containerName="maas-api" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377145 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="69df7514-491c-45c4-bb68-2677a787f0da" containerName="main" Oct 31 20:03:11.377158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.377154 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd23d352-acb4-4b63-addf-478cb67bc561" containerName="istio-proxy" Oct 31 20:03:11.382055 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.382036 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.430788 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.430757 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9902ffd9-serving-cert\"" Oct 31 20:03:11.479712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.479680 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82"] Oct 31 20:03:11.518187 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.518155 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkln\" (UniqueName: \"kubernetes.io/projected/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-kube-api-access-gdkln\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.518366 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.518215 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-maas-api-tls\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.518366 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.518318 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-openshift-service-ca\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.518528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.518510 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-metrics-certs\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.619441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.619408 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-metrics-certs\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.619639 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.619557 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkln\" (UniqueName: \"kubernetes.io/projected/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-kube-api-access-gdkln\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.619639 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.619604 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-maas-api-tls\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.619749 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.619637 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-openshift-service-ca\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.620384 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.620357 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-openshift-service-ca\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.621866 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.621843 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-metrics-certs\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.621947 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.621903 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-maas-api-tls\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.672363 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.672302 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkln\" (UniqueName: \"kubernetes.io/projected/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-kube-api-access-gdkln\") pod \"maas-api-e2e-mt-9902ffd9-6548b5d465-89j82\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:11.695467 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:11.695443 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:12.062845 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:12.062813 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82"] Oct 31 20:03:12.064612 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:03:12.064587 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9afec0_cedd_4041_a4dc_8b11a18f0ff3.slice/crio-8ad5aeabc6da6b8ff6ff1daf0859429219566ce3629ba6cd5c02ba65f32a2f5b WatchSource:0}: Error finding container 8ad5aeabc6da6b8ff6ff1daf0859429219566ce3629ba6cd5c02ba65f32a2f5b: Status 404 returned error can't find the container with id 8ad5aeabc6da6b8ff6ff1daf0859429219566ce3629ba6cd5c02ba65f32a2f5b Oct 31 20:03:12.066219 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:12.066200 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:03:12.420260 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:12.420176 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" event={"ID":"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3","Type":"ContainerStarted","Data":"8ad5aeabc6da6b8ff6ff1daf0859429219566ce3629ba6cd5c02ba65f32a2f5b"} Oct 31 20:03:13.359544 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:13.359457 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm"] Oct 31 20:03:13.359714 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:13.359686 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" containerID="cri-o://b199b2b494f2b40c1f4ebf5082016155d67c1cc304713627e5034c79523bf1d9" gracePeriod=30 Oct 31 20:03:13.502400 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:13.501987 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" event={"ID":"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3","Type":"ContainerStarted","Data":"85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3"} Oct 31 20:03:13.503482 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:13.503426 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:13.612237 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:13.612142 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" podStartSLOduration=1.701670693 podStartE2EDuration="2.612127574s" totalImagesPullingTime="910.456881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:13.6102241 +0000 UTC m=+2764.850447229" watchObservedRunningTime="2026-10-31 20:03:13.612127574 +0000 UTC m=+2764.852350760" Oct 31 20:03:14.999440 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:14.999394 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd"] Oct 31 20:03:15.000091 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.000071 6481 cpu_manager.go:414] "containerMap: removing container" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" Oct 31 20:03:15.000149 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.000092 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9fa3be-2c9a-4960-a69d-fb1e8adcbfbc" containerName="istio-proxy" Oct 31 20:03:15.000232 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.000221 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf1f6bb-a7b0-41d0-a01d-0a988af14ea9" containerName="istio-proxy" Oct 31 20:03:15.003475 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.003458 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.077346 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.077309 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd"] Oct 31 20:03:15.103461 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.103428 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5d59a2-openshift-default-dockercfg-mrsff\"" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190384 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-credential-socket\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190462 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-certs\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190571 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-socket\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190667 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-data\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190793 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-envoy\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190880 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-token\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190907 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-podinfo\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.190956 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istiod-ca-cert\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.191065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.191021 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24jh\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-kube-api-access-c24jh\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292355 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292274 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-data\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292517 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292365 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-envoy\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292517 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292419 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-token\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292517 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292439 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-podinfo\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292628 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292566 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istiod-ca-cert\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292663 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292637 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24jh\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-kube-api-access-c24jh\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292714 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292671 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-data\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292714 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292678 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-credential-socket\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292807 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292749 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-certs\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292862 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292827 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-socket\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.292949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.292915 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-credential-socket\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.293164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.293141 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-certs\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.293239 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.293199 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-socket\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.293368 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.293345 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istiod-ca-cert\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.294776 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.294753 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-envoy\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.294918 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.294899 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-podinfo\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.346325 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.346303 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-token\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.346441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.346357 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24jh\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-kube-api-access-c24jh\") pod \"e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:15.616559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:15.616479 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:16.033348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.033317 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd"] Oct 31 20:03:16.035655 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:03:16.035630 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode481a487_1d1b_42f1_bc66_9638bceeb9c0.slice/crio-5508f014db8d3da2800973e880707e4bc523fccc9abd8d1849ac6de8c496a237 WatchSource:0}: Error finding container 5508f014db8d3da2800973e880707e4bc523fccc9abd8d1849ac6de8c496a237: Status 404 returned error can't find the container with id 5508f014db8d3da2800973e880707e4bc523fccc9abd8d1849ac6de8c496a237 Oct 31 20:03:16.038279 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.038249 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:03:16.038361 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.038314 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:03:16.038361 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.038343 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:03:16.520658 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.520621 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" event={"ID":"e481a487-1d1b-42f1-bc66-9638bceeb9c0","Type":"ContainerStarted","Data":"8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d"} Oct 31 20:03:16.520859 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.520680 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" event={"ID":"e481a487-1d1b-42f1-bc66-9638bceeb9c0","Type":"ContainerStarted","Data":"5508f014db8d3da2800973e880707e4bc523fccc9abd8d1849ac6de8c496a237"} Oct 31 20:03:16.617273 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.617239 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:16.630120 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:16.630070 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" podStartSLOduration=2.630055224 podStartE2EDuration="2.630055224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:16.628861126 +0000 UTC m=+2767.869084265" watchObservedRunningTime="2026-10-31 20:03:16.630055224 +0000 UTC m=+2767.870278352" Oct 31 20:03:17.624195 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:17.624168 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:18.545035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.536559 6481 generic.go:356] "Generic (PLEG): container finished" podID="827ffe64-1796-44a5-b134-99bdca43134f" containerID="b199b2b494f2b40c1f4ebf5082016155d67c1cc304713627e5034c79523bf1d9" exitCode=0 Oct 31 20:03:18.545035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.536962 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" event={"ID":"827ffe64-1796-44a5-b134-99bdca43134f","Type":"ContainerDied","Data":"b199b2b494f2b40c1f4ebf5082016155d67c1cc304713627e5034c79523bf1d9"} Oct 31 20:03:18.545035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.537177 6481 scope.go:122] "RemoveContainer" containerID="2f3753804e83e1e94847988f0647b8159f377fa17e86df84e28be820f584788c" Oct 31 20:03:18.545035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.537604 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:18.545035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.540515 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:03:18.630059 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.629987 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 20:03:18.734766 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.734732 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-envoy\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.734931 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.734832 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-certs\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.734931 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.734888 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/827ffe64-1796-44a5-b134-99bdca43134f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827ffe64-1796-44a5-b134-99bdca43134f-istio-podinfo\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.735074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.734953 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-data\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.735133 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735109 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-certs" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:18.735225 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735202 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-istio-token\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-istio-token\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.735287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735240 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-data" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:18.735287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735258 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-credential-socket\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.735387 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735329 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-socket\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.735438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735392 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-kube-api-access-j8dzk\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-kube-api-access-j8dzk\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.735438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735428 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827ffe64-1796-44a5-b134-99bdca43134f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827ffe64-1796-44a5-b134-99bdca43134f-istiod-ca-cert\") pod \"827ffe64-1796-44a5-b134-99bdca43134f\" (UID: \"827ffe64-1796-44a5-b134-99bdca43134f\") " Oct 31 20:03:18.735549 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735504 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-credential-socket" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:18.735630 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735595 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-socket" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:18.735887 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.735860 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827ffe64-1796-44a5-b134-99bdca43134f-istiod-ca-cert" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:18.736099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.736080 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.736176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.736105 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.736176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.736120 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827ffe64-1796-44a5-b134-99bdca43134f-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.736176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.736133 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.736176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.736148 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.737436 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.737405 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-istio-token" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:18.737559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.737407 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-envoy" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:18.737639 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.737619 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-kube-api-access-j8dzk" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "kube-api-access-j8dzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:18.737695 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.737626 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/827ffe64-1796-44a5-b134-99bdca43134f-istio-podinfo" pod "827ffe64-1796-44a5-b134-99bdca43134f" (UID: "827ffe64-1796-44a5-b134-99bdca43134f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:03:18.837589 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.837553 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827ffe64-1796-44a5-b134-99bdca43134f-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.837589 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.837579 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.837589 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.837589 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8dzk\" (UniqueName: \"kubernetes.io/projected/827ffe64-1796-44a5-b134-99bdca43134f-kube-api-access-j8dzk\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:18.837589 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:18.837598 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827ffe64-1796-44a5-b134-99bdca43134f-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:19.513554 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:19.513519 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:03:19.544210 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:19.544180 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" Oct 31 20:03:19.544210 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:19.544199 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm" event={"ID":"827ffe64-1796-44a5-b134-99bdca43134f","Type":"ContainerDied","Data":"5c705adce8acaa7e6818f0b38304d02b06e24d68c517863b9fb779d766f229d1"} Oct 31 20:03:19.544426 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:19.544253 6481 scope.go:122] "RemoveContainer" containerID="b199b2b494f2b40c1f4ebf5082016155d67c1cc304713627e5034c79523bf1d9" Oct 31 20:03:19.731387 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:19.731341 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm"] Oct 31 20:03:19.740185 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:19.740157 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-650cf3-openshift-default-5f595c7669-45rjm"] Oct 31 20:03:20.880743 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:20.880710 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827ffe64-1796-44a5-b134-99bdca43134f" path="/var/lib/kubelet/pods/827ffe64-1796-44a5-b134-99bdca43134f/volumes" Oct 31 20:03:27.330579 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:03:27.330541 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:03:47.164674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.164641 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn"] Oct 31 20:03:47.165122 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.164836 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" podUID="4b9c28a9-2f92-4649-b02b-da2425aa3f72" containerName="maas-api" containerID="cri-o://bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159" gracePeriod=30 Oct 31 20:03:47.481037 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.481007 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 20:03:47.610015 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.609439 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-metrics-certs\") pod \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " Oct 31 20:03:47.610217 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.610136 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9c28a9-2f92-4649-b02b-da2425aa3f72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9c28a9-2f92-4649-b02b-da2425aa3f72-openshift-service-ca\") pod \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " Oct 31 20:03:47.618016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.610499 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9c28a9-2f92-4649-b02b-da2425aa3f72-openshift-service-ca" pod "4b9c28a9-2f92-4649-b02b-da2425aa3f72" (UID: "4b9c28a9-2f92-4649-b02b-da2425aa3f72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:47.618016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.610577 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls\") pod \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " Oct 31 20:03:47.618016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.611123 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9c28a9-2f92-4649-b02b-da2425aa3f72-kube-api-access-zsfbd\" (UniqueName: \"kubernetes.io/projected/4b9c28a9-2f92-4649-b02b-da2425aa3f72-kube-api-access-zsfbd\") pod \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\" (UID: \"4b9c28a9-2f92-4649-b02b-da2425aa3f72\") " Oct 31 20:03:47.618016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.611687 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9c28a9-2f92-4649-b02b-da2425aa3f72-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:47.621287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.621233 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9c28a9-2f92-4649-b02b-da2425aa3f72-kube-api-access-zsfbd" pod "4b9c28a9-2f92-4649-b02b-da2425aa3f72" (UID: "4b9c28a9-2f92-4649-b02b-da2425aa3f72"). InnerVolumeSpecName "kube-api-access-zsfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:47.632359 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.631764 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls" pod "4b9c28a9-2f92-4649-b02b-da2425aa3f72" (UID: "4b9c28a9-2f92-4649-b02b-da2425aa3f72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:47.632624 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.632582 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-metrics-certs" pod "4b9c28a9-2f92-4649-b02b-da2425aa3f72" (UID: "4b9c28a9-2f92-4649-b02b-da2425aa3f72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:47.731020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.719374 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:47.731020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.719407 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b9c28a9-2f92-4649-b02b-da2425aa3f72-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:47.731020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.719423 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsfbd\" (UniqueName: \"kubernetes.io/projected/4b9c28a9-2f92-4649-b02b-da2425aa3f72-kube-api-access-zsfbd\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:47.741964 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.737176 6481 generic.go:356] "Generic (PLEG): container finished" podID="4b9c28a9-2f92-4649-b02b-da2425aa3f72" containerID="bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159" exitCode=0 Oct 31 20:03:47.741964 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.737277 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" Oct 31 20:03:47.741964 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.737348 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" event={"ID":"4b9c28a9-2f92-4649-b02b-da2425aa3f72","Type":"ContainerDied","Data":"bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159"} Oct 31 20:03:47.741964 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.737396 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn" event={"ID":"4b9c28a9-2f92-4649-b02b-da2425aa3f72","Type":"ContainerDied","Data":"899126f297bce07e47ab8b29985afdc8fcc0f94a6e9517e38002d68d500e569a"} Oct 31 20:03:47.741964 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.737422 6481 scope.go:122] "RemoveContainer" containerID="bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159" Oct 31 20:03:47.760066 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.760048 6481 scope.go:122] "RemoveContainer" containerID="bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159" Oct 31 20:03:47.760389 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:03:47.760367 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159\": container with ID starting with bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159 not found: ID does not exist" containerID="bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159" Oct 31 20:03:47.760479 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.760398 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159"} err="failed to get container status \"bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159\": rpc error: code = NotFound desc = could not find container \"bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159\": container with ID starting with bbd869784f9838f5deee9e340ca4621e0136d57a061bdf39a631c64135287159 not found: ID does not exist" Oct 31 20:03:47.899890 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.899846 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn"] Oct 31 20:03:47.902159 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:47.902120 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5370f5-f87784475-l5rgn"] Oct 31 20:03:48.913659 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:48.908656 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9c28a9-2f92-4649-b02b-da2425aa3f72" path="/var/lib/kubelet/pods/4b9c28a9-2f92-4649-b02b-da2425aa3f72/volumes" Oct 31 20:03:55.438922 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:55.438883 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb"] Oct 31 20:03:55.439450 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:55.439109 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" containerID="cri-o://0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1" gracePeriod=30 Oct 31 20:03:55.636923 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:55.636886 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c"] Oct 31 20:03:55.637131 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:55.637110 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" containerID="cri-o://98653b61eeee2b23288bead99a8b63422cab3a5dd1a84060042732c872f857c4" gracePeriod=30 Oct 31 20:03:56.689276 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:56.689223 6481 patch_prober.go:29] interesting pod/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:03:56.689700 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:56.689284 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:03:57.001808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:57.001774 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr"] Oct 31 20:03:57.001987 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:57.001954 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" containerID="cri-o://62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75" gracePeriod=30 Oct 31 20:03:57.325864 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:03:57.325768 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:03:57.926470 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:57.926430 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj"] Oct 31 20:03:57.927092 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:03:57.926753 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" containerID="cri-o://b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5" gracePeriod=30 Oct 31 20:04:00.771472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.771441 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 20:04:00.832909 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.832874 6481 generic.go:356] "Generic (PLEG): container finished" podID="9bbb9999-8162-4689-9e55-8b39e664322f" containerID="98653b61eeee2b23288bead99a8b63422cab3a5dd1a84060042732c872f857c4" exitCode=0 Oct 31 20:04:00.833080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.832917 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" event={"ID":"9bbb9999-8162-4689-9e55-8b39e664322f","Type":"ContainerDied","Data":"98653b61eeee2b23288bead99a8b63422cab3a5dd1a84060042732c872f857c4"} Oct 31 20:04:00.834973 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.834947 6481 generic.go:356] "Generic (PLEG): container finished" podID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerID="0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1" exitCode=0 Oct 31 20:04:00.835109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.835001 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" Oct 31 20:04:00.835109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.835012 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" event={"ID":"8d75b9d6-fe3f-4b0e-afde-d3995902d013","Type":"ContainerDied","Data":"0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1"} Oct 31 20:04:00.835109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.835059 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb" event={"ID":"8d75b9d6-fe3f-4b0e-afde-d3995902d013","Type":"ContainerDied","Data":"8f7b26119d8464130961c5a4870a8e1ed3c1937d78dd84064c6cb4659986f3f3"} Oct 31 20:04:00.835109 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.835090 6481 scope.go:122] "RemoveContainer" containerID="0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1" Oct 31 20:04:00.848224 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.848195 6481 scope.go:122] "RemoveContainer" containerID="57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138" Oct 31 20:04:00.861890 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.861862 6481 scope.go:122] "RemoveContainer" containerID="0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1" Oct 31 20:04:00.862256 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:00.862233 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1\": container with ID starting with 0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1 not found: ID does not exist" containerID="0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1" Oct 31 20:04:00.862363 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.862262 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1"} err="failed to get container status \"0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1\": rpc error: code = NotFound desc = could not find container \"0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1\": container with ID starting with 0bb3a431602607777844e8f5d357296a5d1f06857c16f56d5024c4508bf64ac1 not found: ID does not exist" Oct 31 20:04:00.862363 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.862280 6481 scope.go:122] "RemoveContainer" containerID="57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138" Oct 31 20:04:00.862627 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:00.862560 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138\": container with ID starting with 57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138 not found: ID does not exist" containerID="57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138" Oct 31 20:04:00.862627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.862597 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138"} err="failed to get container status \"57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138\": rpc error: code = NotFound desc = could not find container \"57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138\": container with ID starting with 57d54d5031294cf403060b6d92649bed74996c1c887ff8a28fb0c26f599d7138 not found: ID does not exist" Oct 31 20:04:00.898599 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.898574 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 20:04:00.953213 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953176 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-envoy\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953237 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-kube-api-access-6ss5m\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-kube-api-access-6ss5m\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953285 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-credential-socket\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953358 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-socket\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953400 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-podinfo\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953717 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953489 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-data\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953717 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953527 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-token\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-token\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953717 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953569 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-certs\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.953717 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953640 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-credential-socket" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.953717 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.953692 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istiod-ca-cert\") pod \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\" (UID: \"8d75b9d6-fe3f-4b0e-afde-d3995902d013\") " Oct 31 20:04:00.954337 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.954128 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-data" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.954337 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.954274 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-socket" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.954533 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.954345 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.954533 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.954368 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:00.954533 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.954465 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istiod-ca-cert" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:00.954930 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.954909 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-certs" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:00.956330 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.956296 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-podinfo" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:00.956424 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.956405 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-token" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:00.956580 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.956557 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-kube-api-access-6ss5m" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "kube-api-access-6ss5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:00.957865 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:00.957844 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-envoy" pod "8d75b9d6-fe3f-4b0e-afde-d3995902d013" (UID: "8d75b9d6-fe3f-4b0e-afde-d3995902d013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.055567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055483 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-kube-api-access-pgx2v\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-kube-api-access-pgx2v\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.055567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055545 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-certs\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.055809 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055610 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-envoy\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.055809 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055660 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbb9999-8162-4689-9e55-8b39e664322f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb9999-8162-4689-9e55-8b39e664322f-istiod-ca-cert\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.055809 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055759 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bbb9999-8162-4689-9e55-8b39e664322f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbb9999-8162-4689-9e55-8b39e664322f-istio-podinfo\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.055809 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055791 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-istio-token\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-istio-token\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.056044 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055847 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-data\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.056044 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055874 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-certs" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.056044 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.055887 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-socket\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.056228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056093 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-socket" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.056228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056094 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-credential-socket\") pod \"9bbb9999-8162-4689-9e55-8b39e664322f\" (UID: \"9bbb9999-8162-4689-9e55-8b39e664322f\") " Oct 31 20:04:01.056228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056114 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbb9999-8162-4689-9e55-8b39e664322f-istiod-ca-cert" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:01.056389 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056295 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-credential-socket" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.056793 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056765 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-data" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056792 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056816 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ss5m\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-kube-api-access-6ss5m\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056833 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056846 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056859 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056873 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056886 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.056904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056900 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d75b9d6-fe3f-4b0e-afde-d3995902d013-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.057389 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056914 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.057389 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056927 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb9999-8162-4689-9e55-8b39e664322f-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.057389 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.056940 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d75b9d6-fe3f-4b0e-afde-d3995902d013-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.058257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.058234 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-envoy" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:01.058348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.058318 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bbb9999-8162-4689-9e55-8b39e664322f-istio-podinfo" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:01.058348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.058338 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-kube-api-access-pgx2v" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "kube-api-access-pgx2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:01.058910 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.058890 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-istio-token" pod "9bbb9999-8162-4689-9e55-8b39e664322f" (UID: "9bbb9999-8162-4689-9e55-8b39e664322f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:01.158429 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.158396 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.158429 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.158426 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgx2v\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-kube-api-access-pgx2v\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.158429 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.158436 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbb9999-8162-4689-9e55-8b39e664322f-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.158682 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.158447 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbb9999-8162-4689-9e55-8b39e664322f-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.158682 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.158457 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbb9999-8162-4689-9e55-8b39e664322f-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.249509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.249463 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb"] Oct 31 20:04:01.256322 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.256291 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7cfafd-openshift-default-59dd46dfc9-bw4kb"] Oct 31 20:04:01.842702 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.842606 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" Oct 31 20:04:01.842702 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.842626 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c" event={"ID":"9bbb9999-8162-4689-9e55-8b39e664322f","Type":"ContainerDied","Data":"902f3e1831e8c87b2e1760e415eb56e0feadb25859fd661713af3568aedc65da"} Oct 31 20:04:01.843260 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.842742 6481 scope.go:122] "RemoveContainer" containerID="98653b61eeee2b23288bead99a8b63422cab3a5dd1a84060042732c872f857c4" Oct 31 20:04:01.995697 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:01.995660 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c"] Oct 31 20:04:02.041778 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.041746 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-cf85c"] Oct 31 20:04:02.282006 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.281964 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 20:04:02.472775 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.472741 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-envoy\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.472957 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.472824 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-token\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-token\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.472957 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.472891 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-credential-socket\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.473089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.472970 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-socket\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.473089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473075 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-podinfo\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.473211 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473142 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-certs\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.473272 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473254 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f34cffd-1e40-424d-9071-99e2595b4ae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f34cffd-1e40-424d-9071-99e2595b4ae6-istiod-ca-cert\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.473342 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473293 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-kube-api-access-4sdml\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-kube-api-access-4sdml\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.473396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473248 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-credential-socket" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:02.473396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473386 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-data\") pod \"3f34cffd-1e40-424d-9071-99e2595b4ae6\" (UID: \"3f34cffd-1e40-424d-9071-99e2595b4ae6\") " Oct 31 20:04:02.473510 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473369 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-socket" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:02.473510 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473408 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-certs" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:02.473741 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473712 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-data" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:02.473741 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473724 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f34cffd-1e40-424d-9071-99e2595b4ae6-istiod-ca-cert" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:02.473955 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473935 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.474034 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473961 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.474034 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.473977 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.474034 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.474006 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.474034 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.474020 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f34cffd-1e40-424d-9071-99e2595b4ae6-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.475275 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.475241 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-token" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:02.475360 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.475311 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-podinfo" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:02.475588 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.475559 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-envoy" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:02.475709 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.475687 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-kube-api-access-4sdml" pod "3f34cffd-1e40-424d-9071-99e2595b4ae6" (UID: "3f34cffd-1e40-424d-9071-99e2595b4ae6"). InnerVolumeSpecName "kube-api-access-4sdml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:02.575203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.575173 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sdml\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-kube-api-access-4sdml\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.575203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.575201 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.575203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.575210 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.575203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.575219 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f34cffd-1e40-424d-9071-99e2595b4ae6-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:02.850523 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.850431 6481 generic.go:356] "Generic (PLEG): container finished" podID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerID="62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75" exitCode=0 Oct 31 20:04:02.850523 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.850477 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" Oct 31 20:04:02.851045 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.850521 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" event={"ID":"3f34cffd-1e40-424d-9071-99e2595b4ae6","Type":"ContainerDied","Data":"62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75"} Oct 31 20:04:02.851045 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.850576 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr" event={"ID":"3f34cffd-1e40-424d-9071-99e2595b4ae6","Type":"ContainerDied","Data":"8aa8f199b519e8445dbd121ed5eac5175a67af82d481ac3811bf24eb92900d2a"} Oct 31 20:04:02.851045 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.850590 6481 scope.go:122] "RemoveContainer" containerID="62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75" Oct 31 20:04:02.861194 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.861116 6481 scope.go:122] "RemoveContainer" containerID="62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75" Oct 31 20:04:02.861390 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:02.861371 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75\": container with ID starting with 62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75 not found: ID does not exist" containerID="62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75" Oct 31 20:04:02.861432 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.861398 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75"} err="failed to get container status \"62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75\": rpc error: code = NotFound desc = could not find container \"62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75\": container with ID starting with 62aba885ed764e39a69c26a1c80866e9b8422b4bfe3270f6c02699b4b342db75 not found: ID does not exist" Oct 31 20:04:02.880348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.880321 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" path="/var/lib/kubelet/pods/8d75b9d6-fe3f-4b0e-afde-d3995902d013/volumes" Oct 31 20:04:02.880795 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:02.880781 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" path="/var/lib/kubelet/pods/9bbb9999-8162-4689-9e55-8b39e664322f/volumes" Oct 31 20:04:03.002765 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.002733 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr"] Oct 31 20:04:03.049197 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.049172 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-662be1-openshift-default-7d999796cf-w8cqr"] Oct 31 20:04:03.196616 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.196591 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 20:04:03.384740 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.384707 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-credential-socket\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.384913 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.384767 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-kube-api-access-pnfbp\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-kube-api-access-pnfbp\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.384913 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.384805 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-socket\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.384913 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.384842 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-data\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.385148 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.384960 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee90507-77f4-497c-b6b6-70b47a953dec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee90507-77f4-497c-b6b6-70b47a953dec-istiod-ca-cert\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.385148 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385036 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-podinfo\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.385148 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385052 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-credential-socket" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:03.385148 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385102 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-certs\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.385364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385158 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-envoy\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.385364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385192 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-token\") pod \"5ee90507-77f4-497c-b6b6-70b47a953dec\" (UID: \"5ee90507-77f4-497c-b6b6-70b47a953dec\") " Oct 31 20:04:03.385364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385219 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-socket" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:03.385364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385230 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-data" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:03.385364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385293 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee90507-77f4-497c-b6b6-70b47a953dec-istiod-ca-cert" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:03.385679 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385653 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-certs" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:03.385817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385729 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.385817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385743 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.385817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385754 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.385817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.385764 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee90507-77f4-497c-b6b6-70b47a953dec-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.387173 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.387152 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-kube-api-access-pnfbp" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "kube-api-access-pnfbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:03.387267 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.387200 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-envoy" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:03.387518 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.387500 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-podinfo" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:03.387575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.387515 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-token" pod "5ee90507-77f4-497c-b6b6-70b47a953dec" (UID: "5ee90507-77f4-497c-b6b6-70b47a953dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:03.487132 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.487103 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.487132 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.487131 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfbp\" (UniqueName: \"kubernetes.io/projected/5ee90507-77f4-497c-b6b6-70b47a953dec-kube-api-access-pnfbp\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.487303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.487141 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.487303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.487150 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.487303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.487159 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee90507-77f4-497c-b6b6-70b47a953dec-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:03.859779 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.859694 6481 generic.go:356] "Generic (PLEG): container finished" podID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerID="b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5" exitCode=0 Oct 31 20:04:03.859779 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.859752 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" Oct 31 20:04:03.860360 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.859775 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" event={"ID":"5ee90507-77f4-497c-b6b6-70b47a953dec","Type":"ContainerDied","Data":"b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5"} Oct 31 20:04:03.860360 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.859843 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj" event={"ID":"5ee90507-77f4-497c-b6b6-70b47a953dec","Type":"ContainerDied","Data":"8dab9ba6c94033a363b2792362383d8d4d93d6db4da3d4077f4fa9b845b614d7"} Oct 31 20:04:03.860360 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.859877 6481 scope.go:122] "RemoveContainer" containerID="b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5" Oct 31 20:04:03.869954 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.869935 6481 scope.go:122] "RemoveContainer" containerID="f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678" Oct 31 20:04:03.878808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.878789 6481 scope.go:122] "RemoveContainer" containerID="b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5" Oct 31 20:04:03.879108 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:03.879088 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5\": container with ID starting with b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5 not found: ID does not exist" containerID="b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5" Oct 31 20:04:03.879164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.879117 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5"} err="failed to get container status \"b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5\": rpc error: code = NotFound desc = could not find container \"b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5\": container with ID starting with b5be238336d6b2a4589b80080388acd23090207fbcdc9b3ebdc9eb4a1d38a4f5 not found: ID does not exist" Oct 31 20:04:03.879164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.879136 6481 scope.go:122] "RemoveContainer" containerID="f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678" Oct 31 20:04:03.879379 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:03.879364 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678\": container with ID starting with f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678 not found: ID does not exist" containerID="f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678" Oct 31 20:04:03.879434 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:03.879381 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678"} err="failed to get container status \"f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678\": rpc error: code = NotFound desc = could not find container \"f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678\": container with ID starting with f80ad2626cc4cd9a8bdd674360343012e635e38f7c5d9c258f4ac9d3708a2678 not found: ID does not exist" Oct 31 20:04:04.013633 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.013598 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj"] Oct 31 20:04:04.059816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.059780 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5370f5-openshift-default-787cd76965-vwzcj"] Oct 31 20:04:04.454182 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454145 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866"] Oct 31 20:04:04.454791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454770 6481 cpu_manager.go:414] "containerMap: removing container" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" Oct 31 20:04:04.454791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454791 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454838 6481 cpu_manager.go:414] "containerMap: removing container" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454847 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454862 6481 cpu_manager.go:414] "containerMap: removing container" podUID="4b9c28a9-2f92-4649-b02b-da2425aa3f72" containerName="maas-api" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454869 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9c28a9-2f92-4649-b02b-da2425aa3f72" containerName="maas-api" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454884 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454890 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454903 6481 cpu_manager.go:414] "containerMap: removing container" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454910 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454921 6481 cpu_manager.go:414] "containerMap: removing container" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454928 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454943 6481 cpu_manager.go:414] "containerMap: removing container" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" Oct 31 20:04:04.454967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454950 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454977 6481 cpu_manager.go:414] "containerMap: removing container" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.454985 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455151 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455164 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9c28a9-2f92-4649-b02b-da2425aa3f72" containerName="maas-api" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455173 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455185 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbb9999-8162-4689-9e55-8b39e664322f" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455195 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455205 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455216 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" Oct 31 20:04:04.455545 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.455227 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d75b9d6-fe3f-4b0e-afde-d3995902d013" containerName="istio-proxy" Oct 31 20:04:04.460721 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.460697 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.498740 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.498709 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8249072-4b53-4698-93f2-3d0e192e20e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.498901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.498787 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.498901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.498840 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trl8r\" (UniqueName: \"kubernetes.io/projected/b8249072-4b53-4698-93f2-3d0e192e20e7-kube-api-access-trl8r\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.499063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.499047 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-metrics-certs\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.509303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.509255 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a89a58-serving-cert\"" Oct 31 20:04:04.511154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.511127 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866"] Oct 31 20:04:04.600009 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.599961 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-metrics-certs\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.600176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.600065 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8249072-4b53-4698-93f2-3d0e192e20e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.600176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.600095 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.600176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.600148 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trl8r\" (UniqueName: \"kubernetes.io/projected/b8249072-4b53-4698-93f2-3d0e192e20e7-kube-api-access-trl8r\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.600332 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:04.600225 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-serving-cert: secret "maas-api-e2e-inf-a-a89a58-serving-cert" not found Oct 31 20:04:04.600332 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:04.600326 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls podName:b8249072-4b53-4698-93f2-3d0e192e20e7 nodeName:}" failed. No retries permitted until 2026-10-31 20:04:05.100304149 +0000 UTC m=+2816.340527276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls") pod "maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" (UID: "b8249072-4b53-4698-93f2-3d0e192e20e7") : secret "maas-api-e2e-inf-a-a89a58-serving-cert" not found Oct 31 20:04:04.600790 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.600769 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8249072-4b53-4698-93f2-3d0e192e20e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.602777 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.602753 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-metrics-certs\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.653211 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.653175 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trl8r\" (UniqueName: \"kubernetes.io/projected/b8249072-4b53-4698-93f2-3d0e192e20e7-kube-api-access-trl8r\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:04.879418 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.879383 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f34cffd-1e40-424d-9071-99e2595b4ae6" path="/var/lib/kubelet/pods/3f34cffd-1e40-424d-9071-99e2595b4ae6/volumes" Oct 31 20:04:04.879875 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:04.879860 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" path="/var/lib/kubelet/pods/5ee90507-77f4-497c-b6b6-70b47a953dec/volumes" Oct 31 20:04:05.120903 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:05.120866 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:05.123285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:05.123263 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:05.376611 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:05.376571 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:05.746738 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:05.745774 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866"] Oct 31 20:04:05.875750 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:05.875715 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" event={"ID":"b8249072-4b53-4698-93f2-3d0e192e20e7","Type":"ContainerStarted","Data":"c397d7600ec695e5ae2c5c2a385bcb675f438208972f8717cd80f5f348960c9d"} Oct 31 20:04:06.883126 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:06.883034 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" event={"ID":"b8249072-4b53-4698-93f2-3d0e192e20e7","Type":"ContainerStarted","Data":"48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645"} Oct 31 20:04:06.883126 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:06.883101 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:06.989519 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:06.989465 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" podStartSLOduration=2.149900993 podStartE2EDuration="2.989451299s" totalImagesPullingTime="839.550306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:06.987158653 +0000 UTC m=+2818.227381782" watchObservedRunningTime="2026-10-31 20:04:06.989451299 +0000 UTC m=+2818.229674465" Oct 31 20:04:07.978904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:07.978869 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2"] Oct 31 20:04:07.979417 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:07.979403 6481 cpu_manager.go:414] "containerMap: removing container" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" Oct 31 20:04:07.979459 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:07.979417 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827ffe64-1796-44a5-b134-99bdca43134f" containerName="istio-proxy" Oct 31 20:04:07.979542 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:07.979533 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee90507-77f4-497c-b6b6-70b47a953dec" containerName="istio-proxy" Oct 31 20:04:07.982579 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:07.982564 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.051767 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.051729 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.051767 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.051769 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4nt\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-kube-api-access-tc4nt\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.051972 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.051841 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.051972 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.051895 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.051972 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.051959 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.052089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.052032 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.052135 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.052123 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.052168 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.052147 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.052201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.052165 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.153636 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.153599 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.153801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.153642 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.153801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.153663 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.153801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.153778 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.153988 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.153822 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4nt\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-kube-api-access-tc4nt\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.153988 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.153873 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.153988 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.153930 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.154221 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.154097 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.154221 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.154181 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.154221 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.154198 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.154378 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.154336 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.154466 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.154440 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.154532 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.154461 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.154666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.154639 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.156094 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.156074 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.156768 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.156741 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.168637 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.168600 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2"] Oct 31 20:04:08.208391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.208353 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.208685 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.208651 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4nt\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-kube-api-access-tc4nt\") pod \"maas-default-gateway-openshift-default-58d898bd55-vpzf2\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.296066 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.295915 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:08.714821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.714795 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2"] Oct 31 20:04:08.716886 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:04:08.716856 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f278bd1_bf4f_4932_a3bd_7f02d90b6173.slice/crio-28838c6f9cd55ce44ab929c005d88ce864bd54c3b15ff51e9856ebe39471ab46 WatchSource:0}: Error finding container 28838c6f9cd55ce44ab929c005d88ce864bd54c3b15ff51e9856ebe39471ab46: Status 404 returned error can't find the container with id 28838c6f9cd55ce44ab929c005d88ce864bd54c3b15ff51e9856ebe39471ab46 Oct 31 20:04:08.719491 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.719461 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:04:08.719563 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.719527 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:04:08.719563 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.719556 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:04:08.903319 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.903271 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" event={"ID":"0f278bd1-bf4f-4932-a3bd-7f02d90b6173","Type":"ContainerStarted","Data":"1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc"} Oct 31 20:04:08.903544 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:08.903336 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" event={"ID":"0f278bd1-bf4f-4932-a3bd-7f02d90b6173","Type":"ContainerStarted","Data":"28838c6f9cd55ce44ab929c005d88ce864bd54c3b15ff51e9856ebe39471ab46"} Oct 31 20:04:09.015143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:09.015020 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podStartSLOduration=2.014983537 podStartE2EDuration="2.014983537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:09.014025577 +0000 UTC m=+2820.254248708" watchObservedRunningTime="2026-10-31 20:04:09.014983537 +0000 UTC m=+2820.255206665" Oct 31 20:04:09.298345 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:09.298252 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:09.298522 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:09.298398 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:09.298522 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:09.298444 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:10.296645 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:10.296604 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:10.297089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:10.296680 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:11.296829 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:11.296796 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:11.297237 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:11.296848 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:12.297276 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:12.297230 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:12.297658 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:12.297304 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:12.901454 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:12.901420 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:04:13.299020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:13.296449 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:13.299020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:13.296513 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:14.307053 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:14.306461 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:14.307053 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:14.306545 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:15.297174 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:15.297131 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:15.297372 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:15.297203 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:16.297128 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:16.297089 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vpzf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 20:04:16.297539 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:16.297150 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 20:04:17.301031 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:17.300959 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:17.301536 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:17.301517 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:17.302245 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:17.302224 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:04:27.326711 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:27.326674 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:04:31.427820 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.427778 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf"] Oct 31 20:04:31.433470 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.433448 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.485056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.485024 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf"] Oct 31 20:04:31.528031 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.527984 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5d59a2-serving-cert\"" Oct 31 20:04:31.572067 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.572027 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.572255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.572198 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.572328 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.572299 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmqqb\" (UniqueName: \"kubernetes.io/projected/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-kube-api-access-qmqqb\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.572467 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.572450 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-metrics-certs\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.691104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.685826 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-metrics-certs\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.691104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.685946 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.691104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.686061 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.691104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.686143 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqqb\" (UniqueName: \"kubernetes.io/projected/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-kube-api-access-qmqqb\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.698024 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.694138 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.700159 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:31.698347 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-serving-cert: secret "maas-api-e2e-inf-b-5d59a2-serving-cert" not found Oct 31 20:04:31.700407 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:31.700391 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls podName:ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57 nodeName:}" failed. No retries permitted until 2026-10-31 20:04:32.200365532 +0000 UTC m=+2843.440588653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls") pod "maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" (UID: "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57") : secret "maas-api-e2e-inf-b-5d59a2-serving-cert" not found Oct 31 20:04:31.728199 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.728163 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-metrics-certs\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:31.739920 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:31.739883 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqqb\" (UniqueName: \"kubernetes.io/projected/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-kube-api-access-qmqqb\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:32.297142 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:32.297097 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:32.299696 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:32.299663 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:32.347312 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:32.347278 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:32.719973 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:32.719951 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf"] Oct 31 20:04:32.722336 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:04:32.722303 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfd8c51_9f8c_454e_89ec_c7f61dbc0d57.slice/crio-6e9341c502437d945e5f2f9420e546f56949d403355691768349c11061acaa9d WatchSource:0}: Error finding container 6e9341c502437d945e5f2f9420e546f56949d403355691768349c11061acaa9d: Status 404 returned error can't find the container with id 6e9341c502437d945e5f2f9420e546f56949d403355691768349c11061acaa9d Oct 31 20:04:33.087034 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:33.086937 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" event={"ID":"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57","Type":"ContainerStarted","Data":"6e9341c502437d945e5f2f9420e546f56949d403355691768349c11061acaa9d"} Oct 31 20:04:34.000634 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.000600 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11074-162327"] Oct 31 20:04:34.004604 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.004579 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:34.021967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.021940 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbwj\" (UniqueName: \"kubernetes.io/projected/dba6e939-a087-4027-807e-df10cafa31ba-kube-api-access-rkbwj\") pod \"test-curl-11074-162327\" (UID: \"dba6e939-a087-4027-807e-df10cafa31ba\") " pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:34.055213 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.055184 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11074-162327"] Oct 31 20:04:34.094196 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.094160 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" event={"ID":"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57","Type":"ContainerStarted","Data":"8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307"} Oct 31 20:04:34.094339 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.094278 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:34.098341 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.098322 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xptzx\"" Oct 31 20:04:34.123918 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.123898 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbwj\" (UniqueName: \"kubernetes.io/projected/dba6e939-a087-4027-807e-df10cafa31ba-kube-api-access-rkbwj\") pod \"test-curl-11074-162327\" (UID: \"dba6e939-a087-4027-807e-df10cafa31ba\") " pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:34.176842 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.176809 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbwj\" (UniqueName: \"kubernetes.io/projected/dba6e939-a087-4027-807e-df10cafa31ba-kube-api-access-rkbwj\") pod \"test-curl-11074-162327\" (UID: \"dba6e939-a087-4027-807e-df10cafa31ba\") " pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:34.214661 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.214607 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" podStartSLOduration=2.280125686 podStartE2EDuration="3.214592925s" totalImagesPullingTime="934.467239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:34.213460134 +0000 UTC m=+2845.453683263" watchObservedRunningTime="2026-10-31 20:04:34.214592925 +0000 UTC m=+2845.454816032" Oct 31 20:04:34.316600 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.316520 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:34.737957 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:34.734138 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11074-162327"] Oct 31 20:04:34.756242 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:04:34.756192 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddba6e939_a087_4027_807e_df10cafa31ba.slice/crio-c21493068be416c91c4ae1da707bd32e22311162a1b113b52b4d97eb522bc922 WatchSource:0}: Error finding container c21493068be416c91c4ae1da707bd32e22311162a1b113b52b4d97eb522bc922: Status 404 returned error can't find the container with id c21493068be416c91c4ae1da707bd32e22311162a1b113b52b4d97eb522bc922 Oct 31 20:04:35.102211 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:35.102120 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11074-162327" event={"ID":"dba6e939-a087-4027-807e-df10cafa31ba","Type":"ContainerStarted","Data":"6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339"} Oct 31 20:04:35.102211 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:35.102173 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11074-162327" event={"ID":"dba6e939-a087-4027-807e-df10cafa31ba","Type":"ContainerStarted","Data":"c21493068be416c91c4ae1da707bd32e22311162a1b113b52b4d97eb522bc922"} Oct 31 20:04:35.208331 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:35.208279 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11074-162327" podStartSLOduration=2.208255326 podStartE2EDuration="2.208255326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:35.205959242 +0000 UTC m=+2846.446182372" watchObservedRunningTime="2026-10-31 20:04:35.208255326 +0000 UTC m=+2846.448478455" Oct 31 20:04:36.597369 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:36.597337 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11074-162327"] Oct 31 20:04:36.600191 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:36.600164 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11074-162327"] Oct 31 20:04:37.128604 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:37.128555 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11074-162327" podUID="dba6e939-a087-4027-807e-df10cafa31ba" containerName="test-curl-11074-162327" containerID="cri-o://6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339" gracePeriod=2 Oct 31 20:04:39.373543 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:39.373512 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:39.391028 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:39.390982 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba6e939-a087-4027-807e-df10cafa31ba-kube-api-access-rkbwj\" (UniqueName: \"kubernetes.io/projected/dba6e939-a087-4027-807e-df10cafa31ba-kube-api-access-rkbwj\") pod \"dba6e939-a087-4027-807e-df10cafa31ba\" (UID: \"dba6e939-a087-4027-807e-df10cafa31ba\") " Oct 31 20:04:39.393016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:39.392975 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba6e939-a087-4027-807e-df10cafa31ba-kube-api-access-rkbwj" pod "dba6e939-a087-4027-807e-df10cafa31ba" (UID: "dba6e939-a087-4027-807e-df10cafa31ba"). InnerVolumeSpecName "kube-api-access-rkbwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:39.492138 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:39.492085 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkbwj\" (UniqueName: \"kubernetes.io/projected/dba6e939-a087-4027-807e-df10cafa31ba-kube-api-access-rkbwj\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.108834 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:40.108803 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:04:40.146861 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:40.146831 6481 generic.go:356] "Generic (PLEG): container finished" podID="dba6e939-a087-4027-807e-df10cafa31ba" containerID="6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339" exitCode=137 Oct 31 20:04:40.147074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:40.146888 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:40.147074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:40.146938 6481 scope.go:122] "RemoveContainer" containerID="6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339" Oct 31 20:04:40.158640 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:40.158577 6481 scope.go:122] "RemoveContainer" containerID="6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339" Oct 31 20:04:40.160904 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:40.160877 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339\": container with ID starting with 6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339 not found: ID does not exist" containerID="6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339" Oct 31 20:04:40.161038 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:40.160913 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339"} err="failed to get container status \"6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339\": rpc error: code = NotFound desc = could not find container \"6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339\": container with ID starting with 6ebb907dd2ae701b07126772dc648d4df6f0095b1e0bc46dab30552ffbbb1339 not found: ID does not exist" Oct 31 20:04:40.263454 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:40.263406 6481 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11074-162327\" is forbidden: User \"system:node:ip-10-0-143-134.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-134.us-west-2.compute.internal' and this object" podUID="dba6e939-a087-4027-807e-df10cafa31ba" pod="opendatahub/test-curl-11074-162327" Oct 31 20:04:40.879655 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:04:40.879621 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba6e939-a087-4027-807e-df10cafa31ba" path="/var/lib/kubelet/pods/dba6e939-a087-4027-807e-df10cafa31ba/volumes" Oct 31 20:04:57.325906 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:04:57.325868 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:05:21.610876 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:21.610792 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn"] Oct 31 20:05:22.036152 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:22.036122 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv"] Oct 31 20:05:22.041048 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:22.041023 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j"] Oct 31 20:05:24.988591 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:24.988551 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96"] Oct 31 20:05:24.989355 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:24.989333 6481 cpu_manager.go:414] "containerMap: removing container" podUID="dba6e939-a087-4027-807e-df10cafa31ba" containerName="test-curl-11074-162327" Oct 31 20:05:24.989467 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:24.989356 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba6e939-a087-4027-807e-df10cafa31ba" containerName="test-curl-11074-162327" Oct 31 20:05:24.989527 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:24.989496 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba6e939-a087-4027-807e-df10cafa31ba" containerName="test-curl-11074-162327" Oct 31 20:05:24.992916 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:24.992894 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028492 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028455 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cd6d98-800c-481c-9685-d737156993bc-istiod-ca-cert\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028653 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028508 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-socket\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028653 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028598 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-credential-socket\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028761 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028667 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-data\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028761 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028695 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cd6d98-800c-481c-9685-d737156993bc-istio-podinfo\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028761 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028729 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-istio-token\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028909 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028805 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-envoy\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.028909 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028891 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-certs\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.029010 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.028967 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktqh\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-kube-api-access-tktqh\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.086310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.086275 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96"] Oct 31 20:05:25.092413 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.092382 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef44dd-openshift-default-dockercfg-5glxc\"" Oct 31 20:05:25.140389 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.140348 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cd6d98-800c-481c-9685-d737156993bc-istiod-ca-cert\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.143457 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.143428 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-socket\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.147871 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.147846 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-credential-socket\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.148439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.141648 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cd6d98-800c-481c-9685-d737156993bc-istiod-ca-cert\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.148601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.143946 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-socket\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.148709 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.148328 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-credential-socket\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.148856 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.148837 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-data\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.149322 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.149219 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-data\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.149484 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.149465 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cd6d98-800c-481c-9685-d737156993bc-istio-podinfo\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.150108 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.150089 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-istio-token\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.150704 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.150667 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-envoy\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.151455 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.151436 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-certs\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.151972 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.151840 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-certs\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.152244 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.152216 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktqh\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-kube-api-access-tktqh\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.155127 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.155095 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cd6d98-800c-481c-9685-d737156993bc-istio-podinfo\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.157556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.157536 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-envoy\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.224818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.224784 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-istio-token\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.243131 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.242943 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktqh\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-kube-api-access-tktqh\") pod \"e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.306964 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.306926 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:25.727016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.726977 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96"] Oct 31 20:05:25.729358 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:05:25.729330 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cd6d98_800c_481c_9685_d737156993bc.slice/crio-6f79f71747f81336052b9a91071189b27a5c6cdb15ba17c8286f4c79e45e03d1 WatchSource:0}: Error finding container 6f79f71747f81336052b9a91071189b27a5c6cdb15ba17c8286f4c79e45e03d1: Status 404 returned error can't find the container with id 6f79f71747f81336052b9a91071189b27a5c6cdb15ba17c8286f4c79e45e03d1 Oct 31 20:05:25.731900 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.731867 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:25.731987 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.731931 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:25.731987 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:25.731959 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:26.517069 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:26.517027 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" event={"ID":"37cd6d98-800c-481c-9685-d737156993bc","Type":"ContainerStarted","Data":"9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c"} Oct 31 20:05:26.517481 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:26.517097 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" event={"ID":"37cd6d98-800c-481c-9685-d737156993bc","Type":"ContainerStarted","Data":"6f79f71747f81336052b9a91071189b27a5c6cdb15ba17c8286f4c79e45e03d1"} Oct 31 20:05:26.675534 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:26.675475 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podStartSLOduration=2.675455963 podStartE2EDuration="2.675455963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:26.67357601 +0000 UTC m=+2897.913799140" watchObservedRunningTime="2026-10-31 20:05:26.675455963 +0000 UTC m=+2897.915679094" Oct 31 20:05:27.307120 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:27.307089 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:27.312968 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:27.312940 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:27.325901 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:27.325872 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:05:27.531601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:27.531567 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:27.532691 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:27.532667 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:05:28.803063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:28.803031 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj"] Oct 31 20:05:29.886616 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:29.886585 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82"] Oct 31 20:05:29.887138 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:29.886839 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" podUID="1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" containerName="maas-api" containerID="cri-o://85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3" gracePeriod=30 Oct 31 20:05:30.169256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.169228 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:05:30.342138 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.342106 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-openshift-service-ca\") pod \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " Oct 31 20:05:30.342331 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.342232 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-kube-api-access-gdkln\" (UniqueName: \"kubernetes.io/projected/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-kube-api-access-gdkln\") pod \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " Oct 31 20:05:30.342331 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.342274 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-metrics-certs\") pod \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " Oct 31 20:05:30.342466 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.342354 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-maas-api-tls\") pod \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\" (UID: \"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3\") " Oct 31 20:05:30.342861 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.342834 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-openshift-service-ca" pod "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" (UID: "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:30.344649 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.344605 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-maas-api-tls" pod "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" (UID: "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:30.345084 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.345057 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-kube-api-access-gdkln" pod "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" (UID: "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3"). InnerVolumeSpecName "kube-api-access-gdkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:30.345178 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.345108 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-metrics-certs" pod "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" (UID: "1b9afec0-cedd-4041-a4dc-8b11a18f0ff3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:30.444147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.444062 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:30.444147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.444101 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdkln\" (UniqueName: \"kubernetes.io/projected/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-kube-api-access-gdkln\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:30.444147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.444117 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:30.444147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.444130 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:30.568540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.568502 6481 generic.go:356] "Generic (PLEG): container finished" podID="1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" containerID="85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3" exitCode=0 Oct 31 20:05:30.568756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.568537 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" event={"ID":"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3","Type":"ContainerDied","Data":"85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3"} Oct 31 20:05:30.568756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.568565 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" Oct 31 20:05:30.568756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.568610 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82" event={"ID":"1b9afec0-cedd-4041-a4dc-8b11a18f0ff3","Type":"ContainerDied","Data":"8ad5aeabc6da6b8ff6ff1daf0859429219566ce3629ba6cd5c02ba65f32a2f5b"} Oct 31 20:05:30.568756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.568644 6481 scope.go:122] "RemoveContainer" containerID="85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3" Oct 31 20:05:30.583398 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.583374 6481 scope.go:122] "RemoveContainer" containerID="85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3" Oct 31 20:05:30.587111 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:30.584215 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3\": container with ID starting with 85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3 not found: ID does not exist" containerID="85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3" Oct 31 20:05:30.587111 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.584257 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3"} err="failed to get container status \"85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3\": rpc error: code = NotFound desc = could not find container \"85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3\": container with ID starting with 85a6d2d45a3bd0842074dd436275fe579309a1ae25881612386e60988dd166d3 not found: ID does not exist" Oct 31 20:05:30.722944 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.722903 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82"] Oct 31 20:05:30.725391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.725359 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902ffd9-6548b5d465-89j82"] Oct 31 20:05:30.879527 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:30.879494 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" path="/var/lib/kubelet/pods/1b9afec0-cedd-4041-a4dc-8b11a18f0ff3/volumes" Oct 31 20:05:33.804328 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:33.804288 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" podUID="4ad37122-9855-4567-aba6-c00abc4eefb8" containerName="payload-pre-processing" containerID="cri-o://de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4" gracePeriod=25 Oct 31 20:05:36.631491 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:36.631424 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" containerID="cri-o://54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365" gracePeriod=45 Oct 31 20:05:36.904416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:36.904385 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 20:05:37.058040 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.057933 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" containerID="cri-o://5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944" gracePeriod=45 Oct 31 20:05:37.064786 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.064726 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" containerID="cri-o://48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b" gracePeriod=45 Oct 31 20:05:37.065095 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.065054 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032d683e-b88d-49f9-ad6a-1ce237aee20c-tls-certs\" (UniqueName: \"kubernetes.io/secret/032d683e-b88d-49f9-ad6a-1ce237aee20c-tls-certs\") pod \"032d683e-b88d-49f9-ad6a-1ce237aee20c\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " Oct 31 20:05:37.065302 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.065281 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-model-cache\") pod \"032d683e-b88d-49f9-ad6a-1ce237aee20c\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " Oct 31 20:05:37.065447 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.065323 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-tmp-dir\") pod \"032d683e-b88d-49f9-ad6a-1ce237aee20c\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " Oct 31 20:05:37.065447 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.065396 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032d683e-b88d-49f9-ad6a-1ce237aee20c-kube-api-access-nsmm5\" (UniqueName: \"kubernetes.io/projected/032d683e-b88d-49f9-ad6a-1ce237aee20c-kube-api-access-nsmm5\") pod \"032d683e-b88d-49f9-ad6a-1ce237aee20c\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " Oct 31 20:05:37.065595 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.065454 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-dshm\") pod \"032d683e-b88d-49f9-ad6a-1ce237aee20c\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " Oct 31 20:05:37.065595 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.065486 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-home\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-home\") pod \"032d683e-b88d-49f9-ad6a-1ce237aee20c\" (UID: \"032d683e-b88d-49f9-ad6a-1ce237aee20c\") " Oct 31 20:05:37.066613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.066583 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-model-cache" pod "032d683e-b88d-49f9-ad6a-1ce237aee20c" (UID: "032d683e-b88d-49f9-ad6a-1ce237aee20c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.067128 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.067005 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-tmp-dir" pod "032d683e-b88d-49f9-ad6a-1ce237aee20c" (UID: "032d683e-b88d-49f9-ad6a-1ce237aee20c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.068267 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.067810 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-home" pod "032d683e-b88d-49f9-ad6a-1ce237aee20c" (UID: "032d683e-b88d-49f9-ad6a-1ce237aee20c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.071872 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.071839 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-dshm" pod "032d683e-b88d-49f9-ad6a-1ce237aee20c" (UID: "032d683e-b88d-49f9-ad6a-1ce237aee20c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.087522 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.087476 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032d683e-b88d-49f9-ad6a-1ce237aee20c-tls-certs" pod "032d683e-b88d-49f9-ad6a-1ce237aee20c" (UID: "032d683e-b88d-49f9-ad6a-1ce237aee20c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:37.087522 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.087500 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032d683e-b88d-49f9-ad6a-1ce237aee20c-kube-api-access-nsmm5" pod "032d683e-b88d-49f9-ad6a-1ce237aee20c" (UID: "032d683e-b88d-49f9-ad6a-1ce237aee20c"). InnerVolumeSpecName "kube-api-access-nsmm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:37.167730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.167547 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.167730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.167595 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.167730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.167612 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsmm5\" (UniqueName: \"kubernetes.io/projected/032d683e-b88d-49f9-ad6a-1ce237aee20c-kube-api-access-nsmm5\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.167730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.167633 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.167730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.167648 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032d683e-b88d-49f9-ad6a-1ce237aee20c-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.167730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.167659 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032d683e-b88d-49f9-ad6a-1ce237aee20c-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.397057 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.396223 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 20:05:37.432759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.432731 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593425 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-dshm\") pod \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593480 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6f425b-defd-4b57-9389-eea11ec6df18-kube-api-access-77gcf\" (UniqueName: \"kubernetes.io/projected/ae6f425b-defd-4b57-9389-eea11ec6df18-kube-api-access-77gcf\") pod \"ae6f425b-defd-4b57-9389-eea11ec6df18\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593596 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-model-cache\") pod \"ae6f425b-defd-4b57-9389-eea11ec6df18\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593650 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-kube-api-access-4ll4c\" (UniqueName: \"kubernetes.io/projected/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-kube-api-access-4ll4c\") pod \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593716 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tmp-dir\") pod \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593757 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-dshm\") pod \"ae6f425b-defd-4b57-9389-eea11ec6df18\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593793 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6f425b-defd-4b57-9389-eea11ec6df18-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f425b-defd-4b57-9389-eea11ec6df18-tls-certs\") pod \"ae6f425b-defd-4b57-9389-eea11ec6df18\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593855 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tls-certs\") pod \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593891 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-tmp-dir\") pod \"ae6f425b-defd-4b57-9389-eea11ec6df18\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.593947 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-home\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-home\") pod \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.594008 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-home\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-home\") pod \"ae6f425b-defd-4b57-9389-eea11ec6df18\" (UID: \"ae6f425b-defd-4b57-9389-eea11ec6df18\") " Oct 31 20:05:37.597019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.594060 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-model-cache\") pod \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\" (UID: \"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b\") " Oct 31 20:05:37.597798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.597167 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-model-cache" pod "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" (UID: "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.597798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.597633 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-tmp-dir" pod "ae6f425b-defd-4b57-9389-eea11ec6df18" (UID: "ae6f425b-defd-4b57-9389-eea11ec6df18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.598822 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.598025 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-home" pod "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" (UID: "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.598822 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.598395 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tmp-dir" pod "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" (UID: "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.598822 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.598778 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-home" pod "ae6f425b-defd-4b57-9389-eea11ec6df18" (UID: "ae6f425b-defd-4b57-9389-eea11ec6df18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.599593 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.599568 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-model-cache" pod "ae6f425b-defd-4b57-9389-eea11ec6df18" (UID: "ae6f425b-defd-4b57-9389-eea11ec6df18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.600444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.600417 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6f425b-defd-4b57-9389-eea11ec6df18-tls-certs" pod "ae6f425b-defd-4b57-9389-eea11ec6df18" (UID: "ae6f425b-defd-4b57-9389-eea11ec6df18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:37.601236 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.601201 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-dshm" pod "ae6f425b-defd-4b57-9389-eea11ec6df18" (UID: "ae6f425b-defd-4b57-9389-eea11ec6df18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.601236 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.601212 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-dshm" pod "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" (UID: "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:37.602189 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.602153 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-kube-api-access-4ll4c" pod "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" (UID: "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b"). InnerVolumeSpecName "kube-api-access-4ll4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:37.602425 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.602400 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6f425b-defd-4b57-9389-eea11ec6df18-kube-api-access-77gcf" pod "ae6f425b-defd-4b57-9389-eea11ec6df18" (UID: "ae6f425b-defd-4b57-9389-eea11ec6df18"). InnerVolumeSpecName "kube-api-access-77gcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:37.603374 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.603341 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tls-certs" pod "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" (UID: "4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:37.624195 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.624160 6481 generic.go:356] "Generic (PLEG): container finished" podID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerID="54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365" exitCode=0 Oct 31 20:05:37.624348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.624205 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" Oct 31 20:05:37.624348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.624241 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" event={"ID":"032d683e-b88d-49f9-ad6a-1ce237aee20c","Type":"ContainerDied","Data":"54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365"} Oct 31 20:05:37.624348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.624293 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn" event={"ID":"032d683e-b88d-49f9-ad6a-1ce237aee20c","Type":"ContainerDied","Data":"49099f764f833de4c5573ffce2f5478a75286efe04edf28aca218a0bcf772c41"} Oct 31 20:05:37.624348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.624317 6481 scope.go:122] "RemoveContainer" containerID="54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365" Oct 31 20:05:37.626046 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.626019 6481 generic.go:356] "Generic (PLEG): container finished" podID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerID="5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944" exitCode=0 Oct 31 20:05:37.626165 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.626100 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" Oct 31 20:05:37.626243 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.626176 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" event={"ID":"ae6f425b-defd-4b57-9389-eea11ec6df18","Type":"ContainerDied","Data":"5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944"} Oct 31 20:05:37.626243 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.626209 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv" event={"ID":"ae6f425b-defd-4b57-9389-eea11ec6df18","Type":"ContainerDied","Data":"9e7b28c41c8938c21e75e9beb1bac6992b4502bb5f6fe18e4c115bf05c6e8be0"} Oct 31 20:05:37.627959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.627931 6481 generic.go:356] "Generic (PLEG): container finished" podID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerID="48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b" exitCode=0 Oct 31 20:05:37.628089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.627986 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" event={"ID":"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b","Type":"ContainerDied","Data":"48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b"} Oct 31 20:05:37.628089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.628062 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" event={"ID":"4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b","Type":"ContainerDied","Data":"15f163f5b3f7331c216247757b711d0e33a7c4f4ab2192397c6fa65453e6ee7a"} Oct 31 20:05:37.628284 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.628149 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j" Oct 31 20:05:37.645286 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.645265 6481 scope.go:122] "RemoveContainer" containerID="54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365" Oct 31 20:05:37.645659 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:37.645625 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365\": container with ID starting with 54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365 not found: ID does not exist" containerID="54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365" Oct 31 20:05:37.645734 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.645651 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365"} err="failed to get container status \"54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365\": rpc error: code = NotFound desc = could not find container \"54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365\": container with ID starting with 54f8cb7150ecfc881429d972b03ed4fdb58ebd965301d827438d55490fa33365 not found: ID does not exist" Oct 31 20:05:37.645734 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.645670 6481 scope.go:122] "RemoveContainer" containerID="5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696409 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696443 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77gcf\" (UniqueName: \"kubernetes.io/projected/ae6f425b-defd-4b57-9389-eea11ec6df18-kube-api-access-77gcf\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696458 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696473 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ll4c\" (UniqueName: \"kubernetes.io/projected/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-kube-api-access-4ll4c\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696487 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696498 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696509 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6f425b-defd-4b57-9389-eea11ec6df18-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696520 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696535 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696547 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696559 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6f425b-defd-4b57-9389-eea11ec6df18-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.696983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.696573 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:37.702720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.702232 6481 scope.go:122] "RemoveContainer" containerID="5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944" Oct 31 20:05:37.702720 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:37.702606 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944\": container with ID starting with 5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944 not found: ID does not exist" containerID="5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944" Oct 31 20:05:37.702720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.702634 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944"} err="failed to get container status \"5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944\": rpc error: code = NotFound desc = could not find container \"5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944\": container with ID starting with 5e9e5f4cb6dbc4ac924e1864fa1faa731c2f82e625e0693156d0466732291944 not found: ID does not exist" Oct 31 20:05:37.702720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.702653 6481 scope.go:122] "RemoveContainer" containerID="48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b" Oct 31 20:05:37.713814 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.713794 6481 scope.go:122] "RemoveContainer" containerID="48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b" Oct 31 20:05:37.714172 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:37.714147 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b\": container with ID starting with 48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b not found: ID does not exist" containerID="48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b" Oct 31 20:05:37.714267 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.714178 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b"} err="failed to get container status \"48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b\": rpc error: code = NotFound desc = could not find container \"48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b\": container with ID starting with 48b524630a8560795462d79978f4c02448da8323c7da3d042b3355cec13e810b not found: ID does not exist" Oct 31 20:05:37.899014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.893711 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn"] Oct 31 20:05:37.917211 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.913366 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-570915/distinct-w5-570915-kserve-65954f587c-x8scn"] Oct 31 20:05:37.992791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.992739 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s"] Oct 31 20:05:37.993679 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993654 6481 cpu_manager.go:414] "containerMap: removing container" podUID="1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" containerName="maas-api" Oct 31 20:05:37.993800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993683 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" containerName="maas-api" Oct 31 20:05:37.993800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993730 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" Oct 31 20:05:37.993800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993739 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" Oct 31 20:05:37.993800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993762 6481 cpu_manager.go:414] "containerMap: removing container" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" Oct 31 20:05:37.993800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993771 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" Oct 31 20:05:37.993800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993799 6481 cpu_manager.go:414] "containerMap: removing container" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" Oct 31 20:05:37.994165 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993807 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" Oct 31 20:05:37.994165 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993938 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" containerName="main" Oct 31 20:05:37.994165 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993952 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" containerName="main" Oct 31 20:05:37.994165 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993966 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" containerName="main" Oct 31 20:05:37.994165 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.993980 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9afec0-cedd-4041-a4dc-8b11a18f0ff3" containerName="maas-api" Oct 31 20:05:37.999516 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.999485 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv"] Oct 31 20:05:37.999674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:37.999596 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.001539 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.001510 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-2-w1-b5155c-kserve-566d684844-bnsjv"] Oct 31 20:05:38.014269 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014241 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.014411 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014288 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vbd\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-kube-api-access-f9vbd\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.014554 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014530 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.014618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014604 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.014738 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014720 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.014819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014761 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.014925 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014908 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.014979 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.014967 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.015066 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.015030 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.061758 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.061220 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s"] Oct 31 20:05:38.102719 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.102677 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j"] Oct 31 20:05:38.110462 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.109283 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b5155c/distinct-w1-b5155c-kserve-754cf54d67-s5k4j"] Oct 31 20:05:38.115711 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.115683 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.115830 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.115736 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.115830 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.115791 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.115962 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.115936 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116036 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.115976 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116100 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116072 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116127 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vbd\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-kube-api-access-f9vbd\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116215 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116203 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116232 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116251 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116388 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116290 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116659 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116632 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116658 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.116759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.116716 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.119028 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.118976 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.119126 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.119033 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.192680 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.192644 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.192909 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.192879 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vbd\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-kube-api-access-f9vbd\") pod \"maas-default-gateway-openshift-default-58d898bd55-rj66s\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.314441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.314346 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:38.338432 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.338395 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf"] Oct 31 20:05:38.759360 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.759335 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s"] Oct 31 20:05:38.762024 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:05:38.761973 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d8e539_fd47_4c03_8ae3_90ddeb4ad7c7.slice/crio-e466e384cadfac56d404a05f6134fe1472aabeb07ba7f5c513bd8f70d9c79772 WatchSource:0}: Error finding container e466e384cadfac56d404a05f6134fe1472aabeb07ba7f5c513bd8f70d9c79772: Status 404 returned error can't find the container with id e466e384cadfac56d404a05f6134fe1472aabeb07ba7f5c513bd8f70d9c79772 Oct 31 20:05:38.764637 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.764605 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:38.764729 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.764684 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:38.764729 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.764717 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:38.901727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.901682 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032d683e-b88d-49f9-ad6a-1ce237aee20c" path="/var/lib/kubelet/pods/032d683e-b88d-49f9-ad6a-1ce237aee20c/volumes" Oct 31 20:05:38.903280 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.903255 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b" path="/var/lib/kubelet/pods/4ccebcd0-551f-49f9-9fb3-6957b2c0ac1b/volumes" Oct 31 20:05:38.903922 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:38.903906 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6f425b-defd-4b57-9389-eea11ec6df18" path="/var/lib/kubelet/pods/ae6f425b-defd-4b57-9389-eea11ec6df18/volumes" Oct 31 20:05:39.652740 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:39.652698 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" event={"ID":"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7","Type":"ContainerStarted","Data":"f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a"} Oct 31 20:05:39.652900 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:39.652777 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" event={"ID":"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7","Type":"ContainerStarted","Data":"e466e384cadfac56d404a05f6134fe1472aabeb07ba7f5c513bd8f70d9c79772"} Oct 31 20:05:39.821052 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:39.820034 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podStartSLOduration=2.819982957 podStartE2EDuration="2.819982957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:39.81372968 +0000 UTC m=+2911.053952807" watchObservedRunningTime="2026-10-31 20:05:39.819982957 +0000 UTC m=+2911.060206087" Oct 31 20:05:40.316336 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:40.316294 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:40.317929 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:40.317902 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rj66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 20:05:40.318094 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:40.318070 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 20:05:41.305145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:41.305100 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" podUID="4ad37122-9855-4567-aba6-c00abc4eefb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:05:41.315482 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:41.315448 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rj66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 20:05:41.315736 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:41.315709 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 20:05:42.316620 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:42.316585 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rj66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 20:05:42.317150 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:42.316826 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 20:05:43.315600 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:43.315554 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rj66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 20:05:43.315753 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:43.315676 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 20:05:44.315759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:44.315725 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rj66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 20:05:44.316184 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:44.315780 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 20:05:45.315268 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:45.315227 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rj66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 20:05:45.315473 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:45.315305 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 20:05:46.315578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.315541 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rj66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 20:05:46.315981 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.315609 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 20:05:46.662308 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.662227 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq"] Oct 31 20:05:46.679981 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.679952 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.759588 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.759406 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-model-cache\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.759588 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.759539 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed78df0-07da-4e7e-be0c-964f5715ff45-tls-certs\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.760743 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.759659 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-tmp-dir\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.760743 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.759704 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-home\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.760743 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.759818 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbw4\" (UniqueName: \"kubernetes.io/projected/2ed78df0-07da-4e7e-be0c-964f5715ff45-kube-api-access-4jbw4\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.760743 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.759950 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-dshm\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.781501 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.781474 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fd95d7\"/\"kube-root-ca.crt\"" Oct 31 20:05:46.781643 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.781533 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fd95d7\"/\"default-dockercfg-ztbjp\"" Oct 31 20:05:46.781809 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.781791 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fd95d7\"/\"test-model-fd95d7-kserve-self-signed-certs\"" Oct 31 20:05:46.781905 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.781890 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fd95d7\"/\"openshift-service-ca.crt\"" Oct 31 20:05:46.785712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.785686 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq"] Oct 31 20:05:46.865065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.864981 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-model-cache\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.865266 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.865072 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed78df0-07da-4e7e-be0c-964f5715ff45-tls-certs\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.867370 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.866472 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-model-cache\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.873937 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.869884 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed78df0-07da-4e7e-be0c-964f5715ff45-tls-certs\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.873937 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.872032 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-tmp-dir\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.873937 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.872089 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-home\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.873937 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.872150 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbw4\" (UniqueName: \"kubernetes.io/projected/2ed78df0-07da-4e7e-be0c-964f5715ff45-kube-api-access-4jbw4\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.873937 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.872265 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-dshm\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.874649 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.874626 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-home\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.879190 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.879164 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-tmp-dir\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.886985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.882279 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-dshm\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:46.935298 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:46.935275 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbw4\" (UniqueName: \"kubernetes.io/projected/2ed78df0-07da-4e7e-be0c-964f5715ff45-kube-api-access-4jbw4\") pod \"test-model-fd95d7-kserve-6f5b7fd597-2krjq\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:47.004559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.004456 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:47.380269 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.380245 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq"] Oct 31 20:05:47.382064 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:05:47.382041 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ed78df0_07da_4e7e_be0c_964f5715ff45.slice/crio-b1afc74e65a2be69717bf25ac1c806306c62db498992afb632fd35d19745797c WatchSource:0}: Error finding container b1afc74e65a2be69717bf25ac1c806306c62db498992afb632fd35d19745797c: Status 404 returned error can't find the container with id b1afc74e65a2be69717bf25ac1c806306c62db498992afb632fd35d19745797c Oct 31 20:05:47.653057 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.652945 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:47.653377 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.653345 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:47.654699 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.654675 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:05:47.735367 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.735330 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" event={"ID":"2ed78df0-07da-4e7e-be0c-964f5715ff45","Type":"ContainerStarted","Data":"03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32"} Oct 31 20:05:47.735528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.735378 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" event={"ID":"2ed78df0-07da-4e7e-be0c-964f5715ff45","Type":"ContainerStarted","Data":"b1afc74e65a2be69717bf25ac1c806306c62db498992afb632fd35d19745797c"} Oct 31 20:05:47.737164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.737127 6481 generic.go:356] "Generic (PLEG): container finished" podID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerID="8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d" exitCode=0 Oct 31 20:05:47.737369 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.737205 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" event={"ID":"e481a487-1d1b-42f1-bc66-9638bceeb9c0","Type":"ContainerDied","Data":"8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d"} Oct 31 20:05:47.737689 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.737674 6481 scope.go:122] "RemoveContainer" containerID="8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d" Oct 31 20:05:47.739039 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.738973 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:47.739157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.739084 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:47.739157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.739118 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:05:47.875016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:47.873750 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" podStartSLOduration=1.8737335480000001 podStartE2EDuration="1.873733548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:47.873214846 +0000 UTC m=+2919.113437976" watchObservedRunningTime="2026-10-31 20:05:47.873733548 +0000 UTC m=+2919.113956677" Oct 31 20:05:48.006820 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:48.006783 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:48.540018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:48.537569 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:05:48.797253 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:48.797164 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" event={"ID":"e481a487-1d1b-42f1-bc66-9638bceeb9c0","Type":"ContainerStarted","Data":"8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223"} Oct 31 20:05:49.617441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:49.617403 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:05:49.618709 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:49.618683 6481 patch_prober.go:29] interesting pod/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 20:05:49.618841 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:49.618735 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 20:05:50.617162 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:50.617124 6481 patch_prober.go:29] interesting pod/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 20:05:50.617386 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:50.617185 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 20:05:51.295858 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:51.295815 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" podUID="4ad37122-9855-4567-aba6-c00abc4eefb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:05:51.617271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:51.617190 6481 patch_prober.go:29] interesting pod/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 20:05:51.617271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:51.617253 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 20:05:52.617075 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:52.617037 6481 patch_prober.go:29] interesting pod/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 20:05:52.617495 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:52.617088 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 20:05:53.361076 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.360987 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" podUID="e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" containerName="main" containerID="cri-o://6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9" gracePeriod=45 Oct 31 20:05:53.617054 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.616951 6481 patch_prober.go:29] interesting pod/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 20:05:53.617221 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.617042 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 20:05:53.660256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.660224 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 20:05:53.808504 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.808460 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-home\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-home\") pod \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " Oct 31 20:05:53.808838 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.808817 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-kube-api-access-jj2rg\" (UniqueName: \"kubernetes.io/projected/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-kube-api-access-jj2rg\") pod \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " Oct 31 20:05:53.809050 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.809030 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-dshm\") pod \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " Oct 31 20:05:53.809242 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.809225 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tmp-dir\") pod \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " Oct 31 20:05:53.809421 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.809406 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tls-certs\") pod \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " Oct 31 20:05:53.809556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.809543 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-model-cache\") pod \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\" (UID: \"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0\") " Oct 31 20:05:53.816355 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.816324 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-home" pod "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" (UID: "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:53.816851 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.816833 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tmp-dir" pod "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" (UID: "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:53.819290 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.818051 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-model-cache" pod "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" (UID: "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:53.825081 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.824941 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-dshm" pod "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" (UID: "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:53.825081 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.825037 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tls-certs" pod "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" (UID: "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:53.825393 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.825374 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-kube-api-access-jj2rg" pod "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" (UID: "e2561fe4-513a-4bdf-be15-68dfbbb8dcd0"). InnerVolumeSpecName "kube-api-access-jj2rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:53.835201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.832802 6481 generic.go:356] "Generic (PLEG): container finished" podID="e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" containerID="6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9" exitCode=0 Oct 31 20:05:53.835201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.832878 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" event={"ID":"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0","Type":"ContainerDied","Data":"6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9"} Oct 31 20:05:53.835201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.832908 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" event={"ID":"e2561fe4-513a-4bdf-be15-68dfbbb8dcd0","Type":"ContainerDied","Data":"5e08e9efc51926e0f9b3f9631847943f4c3e20d9b13865f04b813658046b27c3"} Oct 31 20:05:53.835201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.832926 6481 scope.go:122] "RemoveContainer" containerID="6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9" Oct 31 20:05:53.835201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.833088 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf" Oct 31 20:05:53.912159 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.912130 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:53.912159 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.912158 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2rg\" (UniqueName: \"kubernetes.io/projected/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-kube-api-access-jj2rg\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:53.912411 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.912167 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:53.912411 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.912199 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:53.912411 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.912209 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:53.912411 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.912217 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:53.918114 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.918084 6481 scope.go:122] "RemoveContainer" containerID="6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9" Oct 31 20:05:53.918534 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:53.918500 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9\": container with ID starting with 6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9 not found: ID does not exist" containerID="6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9" Oct 31 20:05:53.918632 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:53.918535 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9"} err="failed to get container status \"6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9\": rpc error: code = NotFound desc = could not find container \"6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9\": container with ID starting with 6deeaa7510d5f74bf581a1de06b663c894419d02821ba9e0a43c2b14390ef4f9 not found: ID does not exist" Oct 31 20:05:54.005337 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.004975 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf"] Oct 31 20:05:54.014416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.014381 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9902ffd9/e2e-lifecycle-model-9902ffd9-kserve-5d8b67499c-pzqsf"] Oct 31 20:05:54.048734 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.048704 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 20:05:54.216310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.216283 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad37122-9855-4567-aba6-c00abc4eefb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad37122-9855-4567-aba6-c00abc4eefb8-plugins-config-volume\") pod \"4ad37122-9855-4567-aba6-c00abc4eefb8\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " Oct 31 20:05:54.216490 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.216407 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad37122-9855-4567-aba6-c00abc4eefb8-kube-api-access-9n5jn\" (UniqueName: \"kubernetes.io/projected/4ad37122-9855-4567-aba6-c00abc4eefb8-kube-api-access-9n5jn\") pod \"4ad37122-9855-4567-aba6-c00abc4eefb8\" (UID: \"4ad37122-9855-4567-aba6-c00abc4eefb8\") " Oct 31 20:05:54.216696 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.216671 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad37122-9855-4567-aba6-c00abc4eefb8-plugins-config-volume" pod "4ad37122-9855-4567-aba6-c00abc4eefb8" (UID: "4ad37122-9855-4567-aba6-c00abc4eefb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:54.218405 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.218385 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad37122-9855-4567-aba6-c00abc4eefb8-kube-api-access-9n5jn" pod "4ad37122-9855-4567-aba6-c00abc4eefb8" (UID: "4ad37122-9855-4567-aba6-c00abc4eefb8"). InnerVolumeSpecName "kube-api-access-9n5jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:54.317657 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.317621 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad37122-9855-4567-aba6-c00abc4eefb8-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.317657 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.317651 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n5jn\" (UniqueName: \"kubernetes.io/projected/4ad37122-9855-4567-aba6-c00abc4eefb8-kube-api-access-9n5jn\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.621959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.621889 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:05:54.622350 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.622166 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:05:54.622782 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.622761 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:05:54.841038 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.840974 6481 generic.go:356] "Generic (PLEG): container finished" podID="4ad37122-9855-4567-aba6-c00abc4eefb8" containerID="de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4" exitCode=0 Oct 31 20:05:54.841038 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.841034 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" Oct 31 20:05:54.841252 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.841057 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" event={"ID":"4ad37122-9855-4567-aba6-c00abc4eefb8","Type":"ContainerDied","Data":"de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4"} Oct 31 20:05:54.841252 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.841106 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj" event={"ID":"4ad37122-9855-4567-aba6-c00abc4eefb8","Type":"ContainerDied","Data":"24a26e27e2b462c9d9b49cacbd05dd26a630a47632ab1f6a400fbf539212bf84"} Oct 31 20:05:54.841252 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.841129 6481 scope.go:122] "RemoveContainer" containerID="de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4" Oct 31 20:05:54.851884 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.851862 6481 scope.go:122] "RemoveContainer" containerID="de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4" Oct 31 20:05:54.852199 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:54.852177 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4\": container with ID starting with de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4 not found: ID does not exist" containerID="de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4" Oct 31 20:05:54.852292 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.852205 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4"} err="failed to get container status \"de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4\": rpc error: code = NotFound desc = could not find container \"de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4\": container with ID starting with de944f2c055a9c17da82e49011705ab2c868be19d310af8ab39a0bc8a2e88ee4 not found: ID does not exist" Oct 31 20:05:54.882158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.882085 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" path="/var/lib/kubelet/pods/e2561fe4-513a-4bdf-be15-68dfbbb8dcd0/volumes" Oct 31 20:05:54.968875 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.968846 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj"] Oct 31 20:05:54.978567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:54.978533 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b5155c-5fc9578447-h7btj"] Oct 31 20:05:56.886207 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:56.886174 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad37122-9855-4567-aba6-c00abc4eefb8" path="/var/lib/kubelet/pods/4ad37122-9855-4567-aba6-c00abc4eefb8/volumes" Oct 31 20:05:57.004628 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:57.004588 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:57.017640 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:57.017610 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:05:57.324336 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:05:57.324301 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:05:57.871712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:05:57.871680 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:06:22.641451 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.641407 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl"] Oct 31 20:06:22.642269 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.642247 6481 cpu_manager.go:414] "containerMap: removing container" podUID="4ad37122-9855-4567-aba6-c00abc4eefb8" containerName="payload-pre-processing" Oct 31 20:06:22.642444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.642271 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad37122-9855-4567-aba6-c00abc4eefb8" containerName="payload-pre-processing" Oct 31 20:06:22.642444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.642288 6481 cpu_manager.go:414] "containerMap: removing container" podUID="e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" containerName="main" Oct 31 20:06:22.642444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.642295 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" containerName="main" Oct 31 20:06:22.642444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.642418 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad37122-9855-4567-aba6-c00abc4eefb8" containerName="payload-pre-processing" Oct 31 20:06:22.642444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.642434 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2561fe4-513a-4bdf-be15-68dfbbb8dcd0" containerName="main" Oct 31 20:06:22.647398 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.647376 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.696088 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.696033 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ef44dd-serving-cert\"" Oct 31 20:06:22.746391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.746353 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl"] Oct 31 20:06:22.803869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.803829 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.804104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.804061 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.804200 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.804101 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-metrics-certs\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.804200 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.804179 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9h6j\" (UniqueName: \"kubernetes.io/projected/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-kube-api-access-x9h6j\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.905949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.905867 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.906135 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.905948 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.906135 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.905967 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-metrics-certs\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.906135 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.905984 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9h6j\" (UniqueName: \"kubernetes.io/projected/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-kube-api-access-x9h6j\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.906564 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.906524 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.908384 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.908361 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.908492 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.908389 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-metrics-certs\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.960984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.960942 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9h6j\" (UniqueName: \"kubernetes.io/projected/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-kube-api-access-x9h6j\") pod \"maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:22.961182 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:22.961171 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:23.331503 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:23.331474 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl"] Oct 31 20:06:23.333543 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:06:23.333516 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fbdeaa_939e_4700_bb30_a1a70c5da6bd.slice/crio-01cd3933eb64874b505989f06953ffe20d7b754d879c703851e6a6ee311dff15 WatchSource:0}: Error finding container 01cd3933eb64874b505989f06953ffe20d7b754d879c703851e6a6ee311dff15: Status 404 returned error can't find the container with id 01cd3933eb64874b505989f06953ffe20d7b754d879c703851e6a6ee311dff15 Oct 31 20:06:24.014889 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:24.014855 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" event={"ID":"08fbdeaa-939e-4700-bb30-a1a70c5da6bd","Type":"ContainerStarted","Data":"01cd3933eb64874b505989f06953ffe20d7b754d879c703851e6a6ee311dff15"} Oct 31 20:06:25.022850 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:25.022815 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" event={"ID":"08fbdeaa-939e-4700-bb30-a1a70c5da6bd","Type":"ContainerStarted","Data":"425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e"} Oct 31 20:06:25.023296 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:25.022933 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:25.129271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:25.129207 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" podStartSLOduration=2.229849136 podStartE2EDuration="3.12918929s" totalImagesPullingTime="899.340154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:25.128559997 +0000 UTC m=+2956.368783136" watchObservedRunningTime="2026-10-31 20:06:25.12918929 +0000 UTC m=+2956.369412420" Oct 31 20:06:27.240488 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.240452 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7"] Oct 31 20:06:27.240972 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.240661 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" podUID="ff52ec1e-fa64-4694-8104-83ae79287e85" containerName="maas-api" containerID="cri-o://1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a" gracePeriod=30 Oct 31 20:06:27.330453 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:06:27.330412 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:06:27.548483 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.548450 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 20:06:27.605842 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.605804 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls\") pod \"ff52ec1e-fa64-4694-8104-83ae79287e85\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " Oct 31 20:06:27.606070 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.605888 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff52ec1e-fa64-4694-8104-83ae79287e85-kube-api-access-4zcm9\" (UniqueName: \"kubernetes.io/projected/ff52ec1e-fa64-4694-8104-83ae79287e85-kube-api-access-4zcm9\") pod \"ff52ec1e-fa64-4694-8104-83ae79287e85\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " Oct 31 20:06:27.606070 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.605976 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff52ec1e-fa64-4694-8104-83ae79287e85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff52ec1e-fa64-4694-8104-83ae79287e85-openshift-service-ca\") pod \"ff52ec1e-fa64-4694-8104-83ae79287e85\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " Oct 31 20:06:27.606070 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.606032 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-metrics-certs\") pod \"ff52ec1e-fa64-4694-8104-83ae79287e85\" (UID: \"ff52ec1e-fa64-4694-8104-83ae79287e85\") " Oct 31 20:06:27.606491 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.606453 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff52ec1e-fa64-4694-8104-83ae79287e85-openshift-service-ca" pod "ff52ec1e-fa64-4694-8104-83ae79287e85" (UID: "ff52ec1e-fa64-4694-8104-83ae79287e85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:27.608379 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.608347 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-metrics-certs" pod "ff52ec1e-fa64-4694-8104-83ae79287e85" (UID: "ff52ec1e-fa64-4694-8104-83ae79287e85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:27.608379 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.608366 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls" pod "ff52ec1e-fa64-4694-8104-83ae79287e85" (UID: "ff52ec1e-fa64-4694-8104-83ae79287e85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:27.608531 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.608349 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff52ec1e-fa64-4694-8104-83ae79287e85-kube-api-access-4zcm9" pod "ff52ec1e-fa64-4694-8104-83ae79287e85" (UID: "ff52ec1e-fa64-4694-8104-83ae79287e85"). InnerVolumeSpecName "kube-api-access-4zcm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:27.710556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.710514 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:27.710556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.710558 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zcm9\" (UniqueName: \"kubernetes.io/projected/ff52ec1e-fa64-4694-8104-83ae79287e85-kube-api-access-4zcm9\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:27.710844 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.710576 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff52ec1e-fa64-4694-8104-83ae79287e85-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:27.710844 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:27.710590 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff52ec1e-fa64-4694-8104-83ae79287e85-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:28.041594 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.041557 6481 generic.go:356] "Generic (PLEG): container finished" podID="ff52ec1e-fa64-4694-8104-83ae79287e85" containerID="1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a" exitCode=0 Oct 31 20:06:28.041933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.041618 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" Oct 31 20:06:28.041933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.041644 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" event={"ID":"ff52ec1e-fa64-4694-8104-83ae79287e85","Type":"ContainerDied","Data":"1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a"} Oct 31 20:06:28.041933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.041710 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7" event={"ID":"ff52ec1e-fa64-4694-8104-83ae79287e85","Type":"ContainerDied","Data":"83a857c92fc90b5039d9d6edbb0d1178ff0dc810d4b2254c8a3281487760d9a3"} Oct 31 20:06:28.041933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.041739 6481 scope.go:122] "RemoveContainer" containerID="1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a" Oct 31 20:06:28.051865 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.051846 6481 scope.go:122] "RemoveContainer" containerID="1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a" Oct 31 20:06:28.052131 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:06:28.052111 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a\": container with ID starting with 1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a not found: ID does not exist" containerID="1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a" Oct 31 20:06:28.052187 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.052140 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a"} err="failed to get container status \"1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a\": rpc error: code = NotFound desc = could not find container \"1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a\": container with ID starting with 1dda6c24df36c80a87d9e0b7275d59059d3b424635b37cd6033971bb609fb72a not found: ID does not exist" Oct 31 20:06:28.256578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.256515 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7"] Oct 31 20:06:28.271019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.269882 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b5155c-5bd9cfcb47-jncp7"] Oct 31 20:06:28.880044 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:28.879982 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff52ec1e-fa64-4694-8104-83ae79287e85" path="/var/lib/kubelet/pods/ff52ec1e-fa64-4694-8104-83ae79287e85/volumes" Oct 31 20:06:31.034808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:31.034780 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:06:38.048895 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:38.048863 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46"] Oct 31 20:06:41.717978 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.717940 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz"] Oct 31 20:06:41.718495 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.718482 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ff52ec1e-fa64-4694-8104-83ae79287e85" containerName="maas-api" Oct 31 20:06:41.718539 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.718496 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff52ec1e-fa64-4694-8104-83ae79287e85" containerName="maas-api" Oct 31 20:06:41.718598 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.718590 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff52ec1e-fa64-4694-8104-83ae79287e85" containerName="maas-api" Oct 31 20:06:41.721959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.721941 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.772443 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.772415 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d59a2\"/\"kube-root-ca.crt\"" Oct 31 20:06:41.818476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.818441 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d59a2\"/\"openshift-service-ca.crt\"" Oct 31 20:06:41.818640 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.818576 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d59a2\"/\"default-dockercfg-6bznq\"" Oct 31 20:06:41.818825 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.818797 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d59a2\"/\"test-model-5d59a2-kserve-self-signed-certs\"" Oct 31 20:06:41.821225 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.821204 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz"] Oct 31 20:06:41.829369 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.829342 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-dshm\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.829494 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.829407 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49713925-9c71-440b-94d3-79818cd211f8-tls-certs\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.829494 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.829472 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-model-cache\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.829597 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.829583 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-home\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.829666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.829607 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc259\" (UniqueName: \"kubernetes.io/projected/49713925-9c71-440b-94d3-79818cd211f8-kube-api-access-wc259\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.829666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.829647 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-tmp-dir\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.931497 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.931425 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-home\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.931819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.931782 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc259\" (UniqueName: \"kubernetes.io/projected/49713925-9c71-440b-94d3-79818cd211f8-kube-api-access-wc259\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.932031 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.931981 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-tmp-dir\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.933720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.933678 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-dshm\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.933978 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.933950 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49713925-9c71-440b-94d3-79818cd211f8-tls-certs\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.934145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.934128 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-model-cache\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.934675 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.934626 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-model-cache\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.934844 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.934730 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-home\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.934944 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.933730 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-tmp-dir\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.956815 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.956654 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-dshm\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.956815 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.956685 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49713925-9c71-440b-94d3-79818cd211f8-tls-certs\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:41.993350 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:41.993279 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc259\" (UniqueName: \"kubernetes.io/projected/49713925-9c71-440b-94d3-79818cd211f8-kube-api-access-wc259\") pod \"test-model-5d59a2-kserve-79bdd74775-h6tnz\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:42.037392 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:42.037366 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:42.422234 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:42.422209 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz"] Oct 31 20:06:42.423956 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:06:42.423924 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49713925_9c71_440b_94d3_79818cd211f8.slice/crio-44104c031559cf079bcb09581545016c3e0d05f020eac604bf8aaa1ed1884459 WatchSource:0}: Error finding container 44104c031559cf079bcb09581545016c3e0d05f020eac604bf8aaa1ed1884459: Status 404 returned error can't find the container with id 44104c031559cf079bcb09581545016c3e0d05f020eac604bf8aaa1ed1884459 Oct 31 20:06:43.154849 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:43.154804 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" event={"ID":"49713925-9c71-440b-94d3-79818cd211f8","Type":"ContainerStarted","Data":"5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4"} Oct 31 20:06:43.155347 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:43.154860 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" event={"ID":"49713925-9c71-440b-94d3-79818cd211f8","Type":"ContainerStarted","Data":"44104c031559cf079bcb09581545016c3e0d05f020eac604bf8aaa1ed1884459"} Oct 31 20:06:43.288017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:43.287239 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" podStartSLOduration=2.287220591 podStartE2EDuration="2.287220591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:43.270363091 +0000 UTC m=+2974.510586219" watchObservedRunningTime="2026-10-31 20:06:43.287220591 +0000 UTC m=+2974.527443721" Oct 31 20:06:44.037914 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:44.037872 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:48.997351 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:48.997272 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9"] Oct 31 20:06:48.997723 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:48.997581 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" podUID="4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" containerName="maas-api" containerID="cri-o://c131fde193b4e0a85afb4e45b3ee8ac1ba40ffe0e671ea45de9d80c5c57783e6" gracePeriod=30 Oct 31 20:06:49.246519 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.246366 6481 generic.go:356] "Generic (PLEG): container finished" podID="4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" containerID="c131fde193b4e0a85afb4e45b3ee8ac1ba40ffe0e671ea45de9d80c5c57783e6" exitCode=0 Oct 31 20:06:49.246519 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.246453 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" event={"ID":"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3","Type":"ContainerDied","Data":"c131fde193b4e0a85afb4e45b3ee8ac1ba40ffe0e671ea45de9d80c5c57783e6"} Oct 31 20:06:49.309261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.309237 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 20:06:49.466759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.466719 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-kube-api-access-5c84s\" (UniqueName: \"kubernetes.io/projected/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-kube-api-access-5c84s\") pod \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " Oct 31 20:06:49.467866 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.466924 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-openshift-service-ca\") pod \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " Oct 31 20:06:49.467866 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.466952 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-metrics-certs\") pod \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " Oct 31 20:06:49.467866 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.467097 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls\") pod \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\" (UID: \"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3\") " Oct 31 20:06:49.468117 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.467850 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-openshift-service-ca" pod "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" (UID: "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:49.468898 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.468864 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:49.477059 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.473163 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls" pod "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" (UID: "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:49.482061 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.482031 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-kube-api-access-5c84s" pod "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" (UID: "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3"). InnerVolumeSpecName "kube-api-access-5c84s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:49.482594 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.482565 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-metrics-certs" pod "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" (UID: "4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:49.574943 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.574083 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:49.574943 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.574121 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:49.574943 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:49.574136 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c84s\" (UniqueName: \"kubernetes.io/projected/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3-kube-api-access-5c84s\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:50.254718 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.254653 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" event={"ID":"4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3","Type":"ContainerDied","Data":"306337b133c50705afc6f9008322f3a4eeee850fc0905f5a67bee52e0aac4b5a"} Oct 31 20:06:50.255257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.254750 6481 scope.go:122] "RemoveContainer" containerID="c131fde193b4e0a85afb4e45b3ee8ac1ba40ffe0e671ea45de9d80c5c57783e6" Oct 31 20:06:50.255257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.254680 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9" Oct 31 20:06:50.260953 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.260921 6481 generic.go:356] "Generic (PLEG): container finished" podID="37cd6d98-800c-481c-9685-d737156993bc" containerID="9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c" exitCode=0 Oct 31 20:06:50.261116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.261019 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" event={"ID":"37cd6d98-800c-481c-9685-d737156993bc","Type":"ContainerDied","Data":"9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c"} Oct 31 20:06:50.261441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.261426 6481 scope.go:122] "RemoveContainer" containerID="9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c" Oct 31 20:06:50.267449 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.267256 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:06:50.267449 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.267335 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:06:50.267449 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.267366 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:06:50.407170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.407139 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9"] Oct 31 20:06:50.412576 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.412552 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-570915-57f6bb8fb9-j7rs9"] Oct 31 20:06:50.896515 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:50.896475 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" path="/var/lib/kubelet/pods/4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3/volumes" Oct 31 20:06:51.293322 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:51.293280 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" event={"ID":"37cd6d98-800c-481c-9685-d737156993bc","Type":"ContainerStarted","Data":"25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30"} Oct 31 20:06:51.307212 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:51.307180 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:06:51.307513 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:51.307493 6481 patch_prober.go:29] interesting pod/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 20:06:51.307565 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:51.307545 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 20:06:52.038189 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:52.038080 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:52.053816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:52.053788 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:52.307607 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:52.307522 6481 patch_prober.go:29] interesting pod/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 20:06:52.307607 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:52.307586 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 20:06:52.311383 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:52.311354 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:06:53.082025 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.080273 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" containerID="cri-o://ebd3ee269c92dd74e4391b85b98fcf03c849cfd0c8683b120066e3016e67bb61" gracePeriod=45 Oct 31 20:06:53.312765 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.312348 6481 patch_prober.go:29] interesting pod/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 20:06:53.312765 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.312423 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 20:06:53.319683 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.318891 6481 generic.go:356] "Generic (PLEG): container finished" podID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerID="ebd3ee269c92dd74e4391b85b98fcf03c849cfd0c8683b120066e3016e67bb61" exitCode=0 Oct 31 20:06:53.319683 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.319596 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" event={"ID":"80c67c4b-3638-4895-b372-08d232d4ce2f","Type":"ContainerDied","Data":"ebd3ee269c92dd74e4391b85b98fcf03c849cfd0c8683b120066e3016e67bb61"} Oct 31 20:06:53.448279 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.448248 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 20:06:53.631984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.631887 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c67c4b-3638-4895-b372-08d232d4ce2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/80c67c4b-3638-4895-b372-08d232d4ce2f-tls-certs\") pod \"80c67c4b-3638-4895-b372-08d232d4ce2f\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " Oct 31 20:06:53.632171 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.631987 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-home\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-home\") pod \"80c67c4b-3638-4895-b372-08d232d4ce2f\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " Oct 31 20:06:53.632171 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632110 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-tmp-dir\") pod \"80c67c4b-3638-4895-b372-08d232d4ce2f\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " Oct 31 20:06:53.632292 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632205 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c67c4b-3638-4895-b372-08d232d4ce2f-kube-api-access-xh769\" (UniqueName: \"kubernetes.io/projected/80c67c4b-3638-4895-b372-08d232d4ce2f-kube-api-access-xh769\") pod \"80c67c4b-3638-4895-b372-08d232d4ce2f\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " Oct 31 20:06:53.632292 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632241 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-model-cache\") pod \"80c67c4b-3638-4895-b372-08d232d4ce2f\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " Oct 31 20:06:53.632402 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632311 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-dshm\") pod \"80c67c4b-3638-4895-b372-08d232d4ce2f\" (UID: \"80c67c4b-3638-4895-b372-08d232d4ce2f\") " Oct 31 20:06:53.632402 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632372 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-tmp-dir" pod "80c67c4b-3638-4895-b372-08d232d4ce2f" (UID: "80c67c4b-3638-4895-b372-08d232d4ce2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:53.632610 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632587 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-model-cache" pod "80c67c4b-3638-4895-b372-08d232d4ce2f" (UID: "80c67c4b-3638-4895-b372-08d232d4ce2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:53.632861 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632842 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-home" pod "80c67c4b-3638-4895-b372-08d232d4ce2f" (UID: "80c67c4b-3638-4895-b372-08d232d4ce2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:53.632963 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632847 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:53.633070 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.632975 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:53.636850 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.635244 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-dshm" pod "80c67c4b-3638-4895-b372-08d232d4ce2f" (UID: "80c67c4b-3638-4895-b372-08d232d4ce2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:53.636850 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.636496 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c67c4b-3638-4895-b372-08d232d4ce2f-tls-certs" pod "80c67c4b-3638-4895-b372-08d232d4ce2f" (UID: "80c67c4b-3638-4895-b372-08d232d4ce2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:53.641425 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.641396 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c67c4b-3638-4895-b372-08d232d4ce2f-kube-api-access-xh769" pod "80c67c4b-3638-4895-b372-08d232d4ce2f" (UID: "80c67c4b-3638-4895-b372-08d232d4ce2f"). InnerVolumeSpecName "kube-api-access-xh769". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:53.734674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.734629 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh769\" (UniqueName: \"kubernetes.io/projected/80c67c4b-3638-4895-b372-08d232d4ce2f-kube-api-access-xh769\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:53.734674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.734677 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:53.734904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.734692 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80c67c4b-3638-4895-b372-08d232d4ce2f-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:53.734904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:53.734722 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80c67c4b-3638-4895-b372-08d232d4ce2f-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:54.308187 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.308141 6481 patch_prober.go:29] interesting pod/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 20:06:54.308378 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.308196 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 20:06:54.325541 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.325515 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" Oct 31 20:06:54.325541 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.325522 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46" event={"ID":"80c67c4b-3638-4895-b372-08d232d4ce2f","Type":"ContainerDied","Data":"b1ea5d7dfadea75a65b95576bf5158a978691362837e93d8e7463487ca64270d"} Oct 31 20:06:54.326118 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.325572 6481 scope.go:122] "RemoveContainer" containerID="ebd3ee269c92dd74e4391b85b98fcf03c849cfd0c8683b120066e3016e67bb61" Oct 31 20:06:54.494030 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.493984 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46"] Oct 31 20:06:54.498055 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.498026 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a4438/distinct-w6-5a4438-kserve-6946c7c7f4-9mq46"] Oct 31 20:06:54.896930 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:54.896893 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" path="/var/lib/kubelet/pods/80c67c4b-3638-4895-b372-08d232d4ce2f/volumes" Oct 31 20:06:55.307288 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:55.307255 6481 patch_prober.go:29] interesting pod/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 20:06:55.307471 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:55.307309 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 20:06:56.312520 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:56.312486 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:06:56.312941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:56.312871 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:06:56.313447 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:56.313430 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:06:57.325534 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:06:57.325498 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:06:58.393254 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:58.393224 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl"] Oct 31 20:06:58.393656 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:06:58.393460 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" containerID="cri-o://399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656" gracePeriod=30 Oct 31 20:07:03.661184 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.661119 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 20:07:03.696261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696230 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-istio-token\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-istio-token\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696423 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696274 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-data\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696423 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696297 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa138a5-163c-48aa-9254-02e50743b0b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa138a5-163c-48aa-9254-02e50743b0b8-istiod-ca-cert\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696423 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696324 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faa138a5-163c-48aa-9254-02e50743b0b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa138a5-163c-48aa-9254-02e50743b0b8-istio-podinfo\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696576 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696467 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-kube-api-access-hxtb4\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-kube-api-access-hxtb4\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696576 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696544 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-data" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:03.696576 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696567 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-certs\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696725 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696618 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-credential-socket\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696725 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696690 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-envoy\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.696827 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696774 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa138a5-163c-48aa-9254-02e50743b0b8-istiod-ca-cert" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:03.696827 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696804 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-socket\") pod \"faa138a5-163c-48aa-9254-02e50743b0b8\" (UID: \"faa138a5-163c-48aa-9254-02e50743b0b8\") " Oct 31 20:07:03.697014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.696971 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-certs" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:03.697205 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.697181 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-credential-socket" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:03.697432 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.697404 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-socket" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:03.697538 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.697516 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.697618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.697540 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.697618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.697557 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa138a5-163c-48aa-9254-02e50743b0b8-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.697618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.697572 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.697618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.697585 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.698780 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.698757 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-istio-token" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:03.699006 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.698966 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-envoy" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:03.699187 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.699166 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faa138a5-163c-48aa-9254-02e50743b0b8-istio-podinfo" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:07:03.699322 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.699302 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-kube-api-access-hxtb4" pod "faa138a5-163c-48aa-9254-02e50743b0b8" (UID: "faa138a5-163c-48aa-9254-02e50743b0b8"). InnerVolumeSpecName "kube-api-access-hxtb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:03.799192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.799161 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa138a5-163c-48aa-9254-02e50743b0b8-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.799192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.799189 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.799192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.799198 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa138a5-163c-48aa-9254-02e50743b0b8-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:03.799423 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:03.799209 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxtb4\" (UniqueName: \"kubernetes.io/projected/faa138a5-163c-48aa-9254-02e50743b0b8-kube-api-access-hxtb4\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:04.387047 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.387006 6481 generic.go:356] "Generic (PLEG): container finished" podID="faa138a5-163c-48aa-9254-02e50743b0b8" containerID="399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656" exitCode=0 Oct 31 20:07:04.387223 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.387064 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" Oct 31 20:07:04.387223 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.387104 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" event={"ID":"faa138a5-163c-48aa-9254-02e50743b0b8","Type":"ContainerDied","Data":"399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656"} Oct 31 20:07:04.387223 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.387157 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl" event={"ID":"faa138a5-163c-48aa-9254-02e50743b0b8","Type":"ContainerDied","Data":"f1329c222a332068fa0ba167295b9359f026107ba7ef42edb0d6de28acf539af"} Oct 31 20:07:04.387223 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.387183 6481 scope.go:122] "RemoveContainer" containerID="399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656" Oct 31 20:07:04.398138 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.398119 6481 scope.go:122] "RemoveContainer" containerID="399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656" Oct 31 20:07:04.398413 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:07:04.398395 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656\": container with ID starting with 399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656 not found: ID does not exist" containerID="399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656" Oct 31 20:07:04.398461 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.398422 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656"} err="failed to get container status \"399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656\": rpc error: code = NotFound desc = could not find container \"399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656\": container with ID starting with 399669db5c2a73840071894c588c51e9a262ceb343d6318bb63dfcbb6e1b9656 not found: ID does not exist" Oct 31 20:07:04.591452 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.591415 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl"] Oct 31 20:07:04.637416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.637348 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-570915-openshift-default-5c8f44f565-p4brl"] Oct 31 20:07:04.878867 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:04.878833 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" path="/var/lib/kubelet/pods/faa138a5-163c-48aa-9254-02e50743b0b8/volumes" Oct 31 20:07:09.858017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:09.857964 6481 scope.go:122] "RemoveContainer" containerID="ffe3c9e0f098d15d503900db1adf7910db78a16d41d7aa399a2b853f122441d0" Oct 31 20:07:27.324706 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:07:27.324672 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:07:55.706172 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:55.706130 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth"] Oct 31 20:07:55.706785 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:55.706343 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" podUID="ece716af-28ae-4bb2-a370-e20910206680" containerName="maas-api" containerID="cri-o://e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae" gracePeriod=30 Oct 31 20:07:56.007406 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.007377 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 20:07:56.130017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.127502 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls\") pod \"ece716af-28ae-4bb2-a370-e20910206680\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " Oct 31 20:07:56.130017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.127580 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-metrics-certs\") pod \"ece716af-28ae-4bb2-a370-e20910206680\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " Oct 31 20:07:56.130017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.127622 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece716af-28ae-4bb2-a370-e20910206680-kube-api-access-glmdq\" (UniqueName: \"kubernetes.io/projected/ece716af-28ae-4bb2-a370-e20910206680-kube-api-access-glmdq\") pod \"ece716af-28ae-4bb2-a370-e20910206680\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " Oct 31 20:07:56.130017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.127805 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece716af-28ae-4bb2-a370-e20910206680-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece716af-28ae-4bb2-a370-e20910206680-openshift-service-ca\") pod \"ece716af-28ae-4bb2-a370-e20910206680\" (UID: \"ece716af-28ae-4bb2-a370-e20910206680\") " Oct 31 20:07:56.136388 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.136172 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece716af-28ae-4bb2-a370-e20910206680-openshift-service-ca" pod "ece716af-28ae-4bb2-a370-e20910206680" (UID: "ece716af-28ae-4bb2-a370-e20910206680"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:56.166096 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.164429 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece716af-28ae-4bb2-a370-e20910206680-kube-api-access-glmdq" pod "ece716af-28ae-4bb2-a370-e20910206680" (UID: "ece716af-28ae-4bb2-a370-e20910206680"). InnerVolumeSpecName "kube-api-access-glmdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:56.168084 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.168034 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls" pod "ece716af-28ae-4bb2-a370-e20910206680" (UID: "ece716af-28ae-4bb2-a370-e20910206680"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:07:56.172035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.171979 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-metrics-certs" pod "ece716af-28ae-4bb2-a370-e20910206680" (UID: "ece716af-28ae-4bb2-a370-e20910206680"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:07:56.230338 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.230264 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece716af-28ae-4bb2-a370-e20910206680-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:56.230338 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.230300 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:56.230338 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.230316 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ece716af-28ae-4bb2-a370-e20910206680-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:56.230338 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.230329 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glmdq\" (UniqueName: \"kubernetes.io/projected/ece716af-28ae-4bb2-a370-e20910206680-kube-api-access-glmdq\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:56.764959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.764921 6481 generic.go:356] "Generic (PLEG): container finished" podID="ece716af-28ae-4bb2-a370-e20910206680" containerID="e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae" exitCode=0 Oct 31 20:07:56.765755 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.765727 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" event={"ID":"ece716af-28ae-4bb2-a370-e20910206680","Type":"ContainerDied","Data":"e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae"} Oct 31 20:07:56.765912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.765897 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" event={"ID":"ece716af-28ae-4bb2-a370-e20910206680","Type":"ContainerDied","Data":"a6005b899db6f278f9c414fdac5bd36da392a0ae368240d72303eb6d2d86b37f"} Oct 31 20:07:56.766050 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.766034 6481 scope.go:122] "RemoveContainer" containerID="e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae" Oct 31 20:07:56.766360 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.766240 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth" Oct 31 20:07:56.783439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.783403 6481 scope.go:122] "RemoveContainer" containerID="e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae" Oct 31 20:07:56.783752 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:07:56.783722 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae\": container with ID starting with e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae not found: ID does not exist" containerID="e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae" Oct 31 20:07:56.783862 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.783757 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae"} err="failed to get container status \"e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae\": rpc error: code = NotFound desc = could not find container \"e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae\": container with ID starting with e319b7b959bf16b732662863f9191fe715caae77bfd6e655b43227cc8bd199ae not found: ID does not exist" Oct 31 20:07:56.921798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.921768 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth"] Oct 31 20:07:56.967829 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:56.967793 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a4438-5448f656c4-xvxth"] Oct 31 20:07:57.328122 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:07:57.328082 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:07:58.879285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:07:58.879248 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece716af-28ae-4bb2-a370-e20910206680" path="/var/lib/kubelet/pods/ece716af-28ae-4bb2-a370-e20910206680/volumes" Oct 31 20:08:07.695119 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:07.695079 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc"] Oct 31 20:08:22.716124 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:22.716035 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" containerID="cri-o://48ca44209cc904dc42e6bd2cab385037bfeb046088db447cd110f7d7f6fc762a" gracePeriod=45 Oct 31 20:08:22.929780 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:22.929749 6481 generic.go:356] "Generic (PLEG): container finished" podID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerID="48ca44209cc904dc42e6bd2cab385037bfeb046088db447cd110f7d7f6fc762a" exitCode=0 Oct 31 20:08:22.929933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:22.929797 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" event={"ID":"90e18125-290b-4f97-acd8-0ab00d15f0db","Type":"ContainerDied","Data":"48ca44209cc904dc42e6bd2cab385037bfeb046088db447cd110f7d7f6fc762a"} Oct 31 20:08:22.983833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:22.983772 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 20:08:23.020915 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.020888 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-home\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-home\") pod \"90e18125-290b-4f97-acd8-0ab00d15f0db\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " Oct 31 20:08:23.021093 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.020965 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e18125-290b-4f97-acd8-0ab00d15f0db-kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/90e18125-290b-4f97-acd8-0ab00d15f0db-kube-api-access-cpkpj\") pod \"90e18125-290b-4f97-acd8-0ab00d15f0db\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " Oct 31 20:08:23.021093 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021034 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e18125-290b-4f97-acd8-0ab00d15f0db-tls-certs\" (UniqueName: \"kubernetes.io/secret/90e18125-290b-4f97-acd8-0ab00d15f0db-tls-certs\") pod \"90e18125-290b-4f97-acd8-0ab00d15f0db\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " Oct 31 20:08:23.021093 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021089 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-model-cache\") pod \"90e18125-290b-4f97-acd8-0ab00d15f0db\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " Oct 31 20:08:23.021270 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021111 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-dshm\") pod \"90e18125-290b-4f97-acd8-0ab00d15f0db\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " Oct 31 20:08:23.021270 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021184 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-tmp-dir\") pod \"90e18125-290b-4f97-acd8-0ab00d15f0db\" (UID: \"90e18125-290b-4f97-acd8-0ab00d15f0db\") " Oct 31 20:08:23.021270 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021218 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-home" pod "90e18125-290b-4f97-acd8-0ab00d15f0db" (UID: "90e18125-290b-4f97-acd8-0ab00d15f0db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.021439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021350 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-model-cache" pod "90e18125-290b-4f97-acd8-0ab00d15f0db" (UID: "90e18125-290b-4f97-acd8-0ab00d15f0db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.021494 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021464 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-tmp-dir" pod "90e18125-290b-4f97-acd8-0ab00d15f0db" (UID: "90e18125-290b-4f97-acd8-0ab00d15f0db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.021638 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021620 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.021708 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021643 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.021708 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.021657 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.023324 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.023296 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e18125-290b-4f97-acd8-0ab00d15f0db-tls-certs" pod "90e18125-290b-4f97-acd8-0ab00d15f0db" (UID: "90e18125-290b-4f97-acd8-0ab00d15f0db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:23.023413 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.023300 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-dshm" pod "90e18125-290b-4f97-acd8-0ab00d15f0db" (UID: "90e18125-290b-4f97-acd8-0ab00d15f0db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:23.023413 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.023303 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e18125-290b-4f97-acd8-0ab00d15f0db-kube-api-access-cpkpj" pod "90e18125-290b-4f97-acd8-0ab00d15f0db" (UID: "90e18125-290b-4f97-acd8-0ab00d15f0db"). InnerVolumeSpecName "kube-api-access-cpkpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:23.123335 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.123300 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpkpj\" (UniqueName: \"kubernetes.io/projected/90e18125-290b-4f97-acd8-0ab00d15f0db-kube-api-access-cpkpj\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.123335 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.123330 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e18125-290b-4f97-acd8-0ab00d15f0db-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.123335 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.123340 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e18125-290b-4f97-acd8-0ab00d15f0db-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:23.936680 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.936656 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" Oct 31 20:08:23.937154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.936692 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc" event={"ID":"90e18125-290b-4f97-acd8-0ab00d15f0db","Type":"ContainerDied","Data":"f5e6a8ab7f08bacf098592d7ab9cedbb585e7af9e7233914518b55c1119deaba"} Oct 31 20:08:23.937154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:23.936741 6481 scope.go:122] "RemoveContainer" containerID="48ca44209cc904dc42e6bd2cab385037bfeb046088db447cd110f7d7f6fc762a" Oct 31 20:08:24.089949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:24.089915 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc"] Oct 31 20:08:24.136261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:24.136225 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-662be1/distinct-w3-662be1-kserve-df786bc88-nn7cc"] Oct 31 20:08:24.879100 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:08:24.879069 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" path="/var/lib/kubelet/pods/90e18125-290b-4f97-acd8-0ab00d15f0db/volumes" Oct 31 20:08:27.390634 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:08:27.389233 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:08:57.325339 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:08:57.325299 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:09:22.444224 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.444185 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l"] Oct 31 20:09:22.444844 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.444389 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" podUID="7eba519c-3954-4e63-ac58-384d264c1494" containerName="maas-api" containerID="cri-o://76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314" gracePeriod=30 Oct 31 20:09:22.748911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.748884 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 20:09:22.905094 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.905048 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls\") pod \"7eba519c-3954-4e63-ac58-384d264c1494\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " Oct 31 20:09:22.905365 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.905341 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eba519c-3954-4e63-ac58-384d264c1494-kube-api-access-59v6h\" (UniqueName: \"kubernetes.io/projected/7eba519c-3954-4e63-ac58-384d264c1494-kube-api-access-59v6h\") pod \"7eba519c-3954-4e63-ac58-384d264c1494\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " Oct 31 20:09:22.905826 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.905804 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-metrics-certs\") pod \"7eba519c-3954-4e63-ac58-384d264c1494\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " Oct 31 20:09:22.906364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.906345 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eba519c-3954-4e63-ac58-384d264c1494-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba519c-3954-4e63-ac58-384d264c1494-openshift-service-ca\") pod \"7eba519c-3954-4e63-ac58-384d264c1494\" (UID: \"7eba519c-3954-4e63-ac58-384d264c1494\") " Oct 31 20:09:22.906950 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.906837 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eba519c-3954-4e63-ac58-384d264c1494-openshift-service-ca" pod "7eba519c-3954-4e63-ac58-384d264c1494" (UID: "7eba519c-3954-4e63-ac58-384d264c1494"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:09:22.907890 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.907864 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba519c-3954-4e63-ac58-384d264c1494-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:22.910214 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.909822 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-metrics-certs" pod "7eba519c-3954-4e63-ac58-384d264c1494" (UID: "7eba519c-3954-4e63-ac58-384d264c1494"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:09:22.910214 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.910151 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eba519c-3954-4e63-ac58-384d264c1494-kube-api-access-59v6h" pod "7eba519c-3954-4e63-ac58-384d264c1494" (UID: "7eba519c-3954-4e63-ac58-384d264c1494"). InnerVolumeSpecName "kube-api-access-59v6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:09:22.914161 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:22.914134 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls" pod "7eba519c-3954-4e63-ac58-384d264c1494" (UID: "7eba519c-3954-4e63-ac58-384d264c1494"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:09:23.013316 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.009737 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:23.013316 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.009779 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eba519c-3954-4e63-ac58-384d264c1494-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:23.013316 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.009794 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59v6h\" (UniqueName: \"kubernetes.io/projected/7eba519c-3954-4e63-ac58-384d264c1494-kube-api-access-59v6h\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:23.335396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.335314 6481 generic.go:356] "Generic (PLEG): container finished" podID="7eba519c-3954-4e63-ac58-384d264c1494" containerID="76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314" exitCode=0 Oct 31 20:09:23.335396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.335361 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" Oct 31 20:09:23.335592 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.335398 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" event={"ID":"7eba519c-3954-4e63-ac58-384d264c1494","Type":"ContainerDied","Data":"76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314"} Oct 31 20:09:23.335592 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.335458 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l" event={"ID":"7eba519c-3954-4e63-ac58-384d264c1494","Type":"ContainerDied","Data":"9d0d444842127a0c21d1ea751cec853f143daced2109ec42ca3654e8e94dcfd2"} Oct 31 20:09:23.335592 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.335475 6481 scope.go:122] "RemoveContainer" containerID="76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314" Oct 31 20:09:23.345169 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.345146 6481 scope.go:122] "RemoveContainer" containerID="76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314" Oct 31 20:09:23.345453 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:09:23.345434 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314\": container with ID starting with 76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314 not found: ID does not exist" containerID="76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314" Oct 31 20:09:23.345501 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.345462 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314"} err="failed to get container status \"76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314\": rpc error: code = NotFound desc = could not find container \"76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314\": container with ID starting with 76de4b2e41cc8077cada153ec29be5fe9c4efdb69dcbd3b71df6640248c5e314 not found: ID does not exist" Oct 31 20:09:23.515186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.515150 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l"] Oct 31 20:09:23.567843 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:23.567807 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-662be1-844cdd6b48-wt76l"] Oct 31 20:09:24.879240 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:09:24.879201 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eba519c-3954-4e63-ac58-384d264c1494" path="/var/lib/kubelet/pods/7eba519c-3954-4e63-ac58-384d264c1494/volumes" Oct 31 20:09:27.326222 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:09:27.326185 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:09:57.327474 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:09:57.327386 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:10:15.773708 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:15.773662 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq"] Oct 31 20:10:27.326276 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:10:27.326238 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:10:30.794411 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:30.794354 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" podUID="2ed78df0-07da-4e7e-be0c-964f5715ff45" containerName="main" containerID="cri-o://03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32" gracePeriod=45 Oct 31 20:10:30.861102 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:30.861062 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" podUID="2ed78df0-07da-4e7e-be0c-964f5715ff45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.137:8000/ready\": dial tcp 10.134.0.137:8000: connect: connection refused" Oct 31 20:10:31.059550 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.059495 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:10:31.149640 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.149610 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ed78df0-07da-4e7e-be0c-964f5715ff45-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed78df0-07da-4e7e-be0c-964f5715ff45-tls-certs\") pod \"2ed78df0-07da-4e7e-be0c-964f5715ff45\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " Oct 31 20:10:31.149790 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.149670 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed78df0-07da-4e7e-be0c-964f5715ff45-kube-api-access-4jbw4\" (UniqueName: \"kubernetes.io/projected/2ed78df0-07da-4e7e-be0c-964f5715ff45-kube-api-access-4jbw4\") pod \"2ed78df0-07da-4e7e-be0c-964f5715ff45\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " Oct 31 20:10:31.149790 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.149760 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-home\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-home\") pod \"2ed78df0-07da-4e7e-be0c-964f5715ff45\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " Oct 31 20:10:31.149913 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.149824 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-dshm\") pod \"2ed78df0-07da-4e7e-be0c-964f5715ff45\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " Oct 31 20:10:31.149913 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.149858 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-model-cache\") pod \"2ed78df0-07da-4e7e-be0c-964f5715ff45\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " Oct 31 20:10:31.150065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.149908 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-tmp-dir\") pod \"2ed78df0-07da-4e7e-be0c-964f5715ff45\" (UID: \"2ed78df0-07da-4e7e-be0c-964f5715ff45\") " Oct 31 20:10:31.150065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.149987 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-home" pod "2ed78df0-07da-4e7e-be0c-964f5715ff45" (UID: "2ed78df0-07da-4e7e-be0c-964f5715ff45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:10:31.150226 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.150186 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-model-cache" pod "2ed78df0-07da-4e7e-be0c-964f5715ff45" (UID: "2ed78df0-07da-4e7e-be0c-964f5715ff45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:10:31.150287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.150258 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-tmp-dir" pod "2ed78df0-07da-4e7e-be0c-964f5715ff45" (UID: "2ed78df0-07da-4e7e-be0c-964f5715ff45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:10:31.150388 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.150370 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:10:31.150439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.150395 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:10:31.150439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.150410 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:10:31.152051 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.152024 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed78df0-07da-4e7e-be0c-964f5715ff45-tls-certs" pod "2ed78df0-07da-4e7e-be0c-964f5715ff45" (UID: "2ed78df0-07da-4e7e-be0c-964f5715ff45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:10:31.152165 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.152102 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed78df0-07da-4e7e-be0c-964f5715ff45-kube-api-access-4jbw4" pod "2ed78df0-07da-4e7e-be0c-964f5715ff45" (UID: "2ed78df0-07da-4e7e-be0c-964f5715ff45"). InnerVolumeSpecName "kube-api-access-4jbw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:10:31.152219 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.152210 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-dshm" pod "2ed78df0-07da-4e7e-be0c-964f5715ff45" (UID: "2ed78df0-07da-4e7e-be0c-964f5715ff45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:10:31.251544 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.251514 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbw4\" (UniqueName: \"kubernetes.io/projected/2ed78df0-07da-4e7e-be0c-964f5715ff45-kube-api-access-4jbw4\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:10:31.251544 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.251540 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed78df0-07da-4e7e-be0c-964f5715ff45-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:10:31.251544 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.251550 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed78df0-07da-4e7e-be0c-964f5715ff45-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:10:31.741309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.741277 6481 generic.go:356] "Generic (PLEG): container finished" podID="2ed78df0-07da-4e7e-be0c-964f5715ff45" containerID="03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32" exitCode=0 Oct 31 20:10:31.741485 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.741319 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" Oct 31 20:10:31.741485 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.741376 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" event={"ID":"2ed78df0-07da-4e7e-be0c-964f5715ff45","Type":"ContainerDied","Data":"03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32"} Oct 31 20:10:31.741485 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.741426 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq" event={"ID":"2ed78df0-07da-4e7e-be0c-964f5715ff45","Type":"ContainerDied","Data":"b1afc74e65a2be69717bf25ac1c806306c62db498992afb632fd35d19745797c"} Oct 31 20:10:31.741485 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.741453 6481 scope.go:122] "RemoveContainer" containerID="03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32" Oct 31 20:10:31.754456 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.754035 6481 scope.go:122] "RemoveContainer" containerID="03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32" Oct 31 20:10:31.754456 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:10:31.754446 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32\": container with ID starting with 03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32 not found: ID does not exist" containerID="03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32" Oct 31 20:10:31.754590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.754469 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32"} err="failed to get container status \"03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32\": rpc error: code = NotFound desc = could not find container \"03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32\": container with ID starting with 03b448583d8af5536c82eeb96ce96c25e5a2d4bc74fecf30b57feb6029b6be32 not found: ID does not exist" Oct 31 20:10:31.908553 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.908519 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq"] Oct 31 20:10:31.959733 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:31.959703 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fd95d7/test-model-fd95d7-kserve-6f5b7fd597-2krjq"] Oct 31 20:10:32.878820 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:32.878783 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed78df0-07da-4e7e-be0c-964f5715ff45" path="/var/lib/kubelet/pods/2ed78df0-07da-4e7e-be0c-964f5715ff45/volumes" Oct 31 20:10:55.297811 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:10:55.297773 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz"] Oct 31 20:10:57.324008 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:10:57.323956 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:11:10.318306 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.318266 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" podUID="49713925-9c71-440b-94d3-79818cd211f8" containerName="main" containerID="cri-o://5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4" gracePeriod=45 Oct 31 20:11:10.586982 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.586908 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:11:10.665602 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.665566 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-tmp-dir\") pod \"49713925-9c71-440b-94d3-79818cd211f8\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " Oct 31 20:11:10.665798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.665626 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49713925-9c71-440b-94d3-79818cd211f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/49713925-9c71-440b-94d3-79818cd211f8-tls-certs\") pod \"49713925-9c71-440b-94d3-79818cd211f8\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " Oct 31 20:11:10.665798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.665721 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-dshm\") pod \"49713925-9c71-440b-94d3-79818cd211f8\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " Oct 31 20:11:10.665798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.665740 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-model-cache\") pod \"49713925-9c71-440b-94d3-79818cd211f8\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " Oct 31 20:11:10.665798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.665793 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49713925-9c71-440b-94d3-79818cd211f8-kube-api-access-wc259\" (UniqueName: \"kubernetes.io/projected/49713925-9c71-440b-94d3-79818cd211f8-kube-api-access-wc259\") pod \"49713925-9c71-440b-94d3-79818cd211f8\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " Oct 31 20:11:10.666038 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.665865 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-home\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-home\") pod \"49713925-9c71-440b-94d3-79818cd211f8\" (UID: \"49713925-9c71-440b-94d3-79818cd211f8\") " Oct 31 20:11:10.666038 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.665868 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-tmp-dir" pod "49713925-9c71-440b-94d3-79818cd211f8" (UID: "49713925-9c71-440b-94d3-79818cd211f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:10.666176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.666110 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-model-cache" pod "49713925-9c71-440b-94d3-79818cd211f8" (UID: "49713925-9c71-440b-94d3-79818cd211f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:10.666176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.666130 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-home" pod "49713925-9c71-440b-94d3-79818cd211f8" (UID: "49713925-9c71-440b-94d3-79818cd211f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:10.666386 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.666362 6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-model-cache\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:10.666386 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.666384 6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-home\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:10.666551 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.666398 6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-tmp-dir\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:10.667972 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.667950 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-dshm" pod "49713925-9c71-440b-94d3-79818cd211f8" (UID: "49713925-9c71-440b-94d3-79818cd211f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:11:10.668078 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.668041 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49713925-9c71-440b-94d3-79818cd211f8-kube-api-access-wc259" pod "49713925-9c71-440b-94d3-79818cd211f8" (UID: "49713925-9c71-440b-94d3-79818cd211f8"). InnerVolumeSpecName "kube-api-access-wc259". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:11:10.668078 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.668067 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49713925-9c71-440b-94d3-79818cd211f8-tls-certs" pod "49713925-9c71-440b-94d3-79818cd211f8" (UID: "49713925-9c71-440b-94d3-79818cd211f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:11:10.767676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.767643 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc259\" (UniqueName: \"kubernetes.io/projected/49713925-9c71-440b-94d3-79818cd211f8-kube-api-access-wc259\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:10.767676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.767673 6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49713925-9c71-440b-94d3-79818cd211f8-tls-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:10.767676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:10.767683 6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49713925-9c71-440b-94d3-79818cd211f8-dshm\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:11:11.054517 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.054482 6481 generic.go:356] "Generic (PLEG): container finished" podID="49713925-9c71-440b-94d3-79818cd211f8" containerID="5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4" exitCode=0 Oct 31 20:11:11.054697 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.054524 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" Oct 31 20:11:11.054697 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.054563 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" event={"ID":"49713925-9c71-440b-94d3-79818cd211f8","Type":"ContainerDied","Data":"5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4"} Oct 31 20:11:11.054697 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.054608 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz" event={"ID":"49713925-9c71-440b-94d3-79818cd211f8","Type":"ContainerDied","Data":"44104c031559cf079bcb09581545016c3e0d05f020eac604bf8aaa1ed1884459"} Oct 31 20:11:11.054697 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.054625 6481 scope.go:122] "RemoveContainer" containerID="5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4" Oct 31 20:11:11.063842 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.063824 6481 scope.go:122] "RemoveContainer" containerID="5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4" Oct 31 20:11:11.064136 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:11:11.064116 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4\": container with ID starting with 5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4 not found: ID does not exist" containerID="5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4" Oct 31 20:11:11.064188 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.064144 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4"} err="failed to get container status \"5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4\": rpc error: code = NotFound desc = could not find container \"5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4\": container with ID starting with 5c27e475e7f1ba830364bebcecdecfe8de1639ef35b7aaa13c9ac279eb8983a4 not found: ID does not exist" Oct 31 20:11:11.223065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.223027 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz"] Oct 31 20:11:11.274113 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:11.274075 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5d59a2/test-model-5d59a2-kserve-79bdd74775-h6tnz"] Oct 31 20:11:12.879267 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:12.879231 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49713925-9c71-440b-94d3-79818cd211f8" path="/var/lib/kubelet/pods/49713925-9c71-440b-94d3-79818cd211f8/volumes" Oct 31 20:11:27.324941 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:11:27.324857 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:11:56.697938 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:56.697907 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl"] Oct 31 20:11:56.700536 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:11:56.698137 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" containerID="cri-o://d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37" gracePeriod=30 Oct 31 20:11:57.324546 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:11:57.324506 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:12:01.958698 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:01.958672 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:12:02.110715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110633 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-credential-socket\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.110715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110679 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-kube-api-access-hm5bl\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-kube-api-access-hm5bl\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.110966 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110719 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-podinfo\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.110966 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110797 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-socket\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.110966 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110850 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-data\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.110966 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110884 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-certs\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.110966 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110946 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-credential-socket" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.111291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.110983 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-token\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.111291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111057 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istiod-ca-cert\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.111291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111085 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-envoy\") pod \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\" (UID: \"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a\") " Oct 31 20:12:02.111291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111182 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-certs" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.111291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111193 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-data" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.111291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111208 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-socket" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.111618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111478 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istiod-ca-cert" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:02.111618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111547 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.111618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111569 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.111618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111584 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.111618 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.111598 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.113091 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.113064 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-podinfo" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:12:02.113211 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.113197 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-envoy" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:02.113599 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.113576 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-kube-api-access-hm5bl" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "kube-api-access-hm5bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:02.113693 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.113601 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-token" pod "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" (UID: "0d6ca183-e38a-43ba-bd2e-8d444dbcc29a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:02.213201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.213166 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.213201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.213195 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.213201 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.213206 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm5bl\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-kube-api-access-hm5bl\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.213428 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.213216 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.213428 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.213230 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:02.335068 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.335033 6481 generic.go:356] "Generic (PLEG): container finished" podID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerID="d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37" exitCode=0 Oct 31 20:12:02.335246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.335088 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" event={"ID":"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a","Type":"ContainerDied","Data":"d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37"} Oct 31 20:12:02.335246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.335092 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" Oct 31 20:12:02.335246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.335131 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl" event={"ID":"0d6ca183-e38a-43ba-bd2e-8d444dbcc29a","Type":"ContainerDied","Data":"6c65e6cfaa3b453a19f6d3e02dae3e71450981edb943594bcd31c96754499ed0"} Oct 31 20:12:02.335246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.335151 6481 scope.go:122] "RemoveContainer" containerID="d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37" Oct 31 20:12:02.345387 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.345365 6481 scope.go:122] "RemoveContainer" containerID="d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37" Oct 31 20:12:02.345674 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:02.345653 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37\": container with ID starting with d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37 not found: ID does not exist" containerID="d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37" Oct 31 20:12:02.345740 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.345689 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37"} err="failed to get container status \"d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37\": rpc error: code = NotFound desc = could not find container \"d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37\": container with ID starting with d1a6fe236fc0b8a084373a34225d389340cb01a7270c2f41845f468eb97b0e37 not found: ID does not exist" Oct 31 20:12:02.502311 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.502281 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl"] Oct 31 20:12:02.505362 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.505336 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fd95d7-openshift-default-bfd75857-qwvvl"] Oct 31 20:12:02.879437 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:02.879358 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" path="/var/lib/kubelet/pods/0d6ca183-e38a-43ba-bd2e-8d444dbcc29a/volumes" Oct 31 20:12:19.381985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.381947 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866"] Oct 31 20:12:19.382474 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.382214 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" podUID="b8249072-4b53-4698-93f2-3d0e192e20e7" containerName="maas-api" containerID="cri-o://48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645" gracePeriod=30 Oct 31 20:12:19.668672 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.668638 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:12:19.823089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.823054 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8249072-4b53-4698-93f2-3d0e192e20e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8249072-4b53-4698-93f2-3d0e192e20e7-openshift-service-ca\") pod \"b8249072-4b53-4698-93f2-3d0e192e20e7\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " Oct 31 20:12:19.823261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.823110 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-metrics-certs\") pod \"b8249072-4b53-4698-93f2-3d0e192e20e7\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " Oct 31 20:12:19.823261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.823225 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls\") pod \"b8249072-4b53-4698-93f2-3d0e192e20e7\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " Oct 31 20:12:19.823261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.823256 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8249072-4b53-4698-93f2-3d0e192e20e7-kube-api-access-trl8r\" (UniqueName: \"kubernetes.io/projected/b8249072-4b53-4698-93f2-3d0e192e20e7-kube-api-access-trl8r\") pod \"b8249072-4b53-4698-93f2-3d0e192e20e7\" (UID: \"b8249072-4b53-4698-93f2-3d0e192e20e7\") " Oct 31 20:12:19.823474 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.823432 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8249072-4b53-4698-93f2-3d0e192e20e7-openshift-service-ca" pod "b8249072-4b53-4698-93f2-3d0e192e20e7" (UID: "b8249072-4b53-4698-93f2-3d0e192e20e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:19.824197 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.824166 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8249072-4b53-4698-93f2-3d0e192e20e7-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:19.825431 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.825401 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls" pod "b8249072-4b53-4698-93f2-3d0e192e20e7" (UID: "b8249072-4b53-4698-93f2-3d0e192e20e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:19.825783 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.825765 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-metrics-certs" pod "b8249072-4b53-4698-93f2-3d0e192e20e7" (UID: "b8249072-4b53-4698-93f2-3d0e192e20e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:19.825856 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.825816 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8249072-4b53-4698-93f2-3d0e192e20e7-kube-api-access-trl8r" pod "b8249072-4b53-4698-93f2-3d0e192e20e7" (UID: "b8249072-4b53-4698-93f2-3d0e192e20e7"). InnerVolumeSpecName "kube-api-access-trl8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:19.942701 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.941355 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:19.942701 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.941397 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8249072-4b53-4698-93f2-3d0e192e20e7-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:19.942701 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:19.941413 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trl8r\" (UniqueName: \"kubernetes.io/projected/b8249072-4b53-4698-93f2-3d0e192e20e7-kube-api-access-trl8r\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:20.430244 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.430205 6481 generic.go:356] "Generic (PLEG): container finished" podID="b8249072-4b53-4698-93f2-3d0e192e20e7" containerID="48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645" exitCode=0 Oct 31 20:12:20.430635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.430251 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" Oct 31 20:12:20.430635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.430292 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" event={"ID":"b8249072-4b53-4698-93f2-3d0e192e20e7","Type":"ContainerDied","Data":"48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645"} Oct 31 20:12:20.430635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.430345 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866" event={"ID":"b8249072-4b53-4698-93f2-3d0e192e20e7","Type":"ContainerDied","Data":"c397d7600ec695e5ae2c5c2a385bcb675f438208972f8717cd80f5f348960c9d"} Oct 31 20:12:20.430635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.430371 6481 scope.go:122] "RemoveContainer" containerID="48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645" Oct 31 20:12:20.440292 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.440270 6481 scope.go:122] "RemoveContainer" containerID="48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645" Oct 31 20:12:20.440601 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:20.440574 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645\": container with ID starting with 48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645 not found: ID does not exist" containerID="48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645" Oct 31 20:12:20.440709 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.440610 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645"} err="failed to get container status \"48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645\": rpc error: code = NotFound desc = could not find container \"48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645\": container with ID starting with 48126ecaebe0c7f5208d5f8af27ff6798516a27857f99c0e190873fa1d65a645 not found: ID does not exist" Oct 31 20:12:20.598925 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.598890 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866"] Oct 31 20:12:20.650248 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.650213 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a89a58-5b56fd646b-8n866"] Oct 31 20:12:20.879526 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:20.879490 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8249072-4b53-4698-93f2-3d0e192e20e7" path="/var/lib/kubelet/pods/b8249072-4b53-4698-93f2-3d0e192e20e7/volumes" Oct 31 20:12:27.324957 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:27.324915 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:12:30.933732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:30.933692 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf"] Oct 31 20:12:30.934212 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:30.933924 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" podUID="ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" containerName="maas-api" containerID="cri-o://8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307" gracePeriod=30 Oct 31 20:12:31.225373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.225348 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:12:31.304907 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.304873 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-metrics-certs\") pod \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " Oct 31 20:12:31.305113 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.305047 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-openshift-service-ca\") pod \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " Oct 31 20:12:31.305113 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.305098 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls\") pod \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " Oct 31 20:12:31.305215 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.305126 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-kube-api-access-qmqqb\" (UniqueName: \"kubernetes.io/projected/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-kube-api-access-qmqqb\") pod \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\" (UID: \"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57\") " Oct 31 20:12:31.305418 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.305392 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-openshift-service-ca" pod "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" (UID: "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:31.305674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.305642 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:31.307193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.307162 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-metrics-certs" pod "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" (UID: "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:31.307281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.307189 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls" pod "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" (UID: "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:31.307281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.307252 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-kube-api-access-qmqqb" pod "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" (UID: "ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57"). InnerVolumeSpecName "kube-api-access-qmqqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:31.407871 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.407834 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:31.407871 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.407875 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmqqb\" (UniqueName: \"kubernetes.io/projected/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-kube-api-access-qmqqb\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:31.408134 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.407891 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:31.504027 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.503975 6481 generic.go:356] "Generic (PLEG): container finished" podID="ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" containerID="8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307" exitCode=0 Oct 31 20:12:31.504296 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.504078 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" event={"ID":"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57","Type":"ContainerDied","Data":"8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307"} Oct 31 20:12:31.504296 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.504094 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" Oct 31 20:12:31.504296 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.504139 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf" event={"ID":"ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57","Type":"ContainerDied","Data":"6e9341c502437d945e5f2f9420e546f56949d403355691768349c11061acaa9d"} Oct 31 20:12:31.504296 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.504162 6481 scope.go:122] "RemoveContainer" containerID="8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307" Oct 31 20:12:31.515307 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.515264 6481 scope.go:122] "RemoveContainer" containerID="8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307" Oct 31 20:12:31.515640 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:31.515585 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307\": container with ID starting with 8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307 not found: ID does not exist" containerID="8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307" Oct 31 20:12:31.515734 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.515650 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307"} err="failed to get container status \"8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307\": rpc error: code = NotFound desc = could not find container \"8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307\": container with ID starting with 8f108c294f542fe51476914e047618ee818cea436f977140ba608d2beef6b307 not found: ID does not exist" Oct 31 20:12:31.672040 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.672008 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf"] Oct 31 20:12:31.675331 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:31.675309 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d59a2-7b7877cdb7-9mxnf"] Oct 31 20:12:32.879467 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:32.879426 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" path="/var/lib/kubelet/pods/ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57/volumes" Oct 31 20:12:39.969728 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:39.969694 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd"] Oct 31 20:12:39.970186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:39.969911 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" containerID="cri-o://8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223" gracePeriod=30 Oct 31 20:12:45.238430 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.238372 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:12:45.398726 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.398688 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-socket\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.398911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.398768 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istiod-ca-cert\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.398911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.398804 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-certs\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.398911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.398834 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-envoy\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.398911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.398891 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-kube-api-access-c24jh\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-kube-api-access-c24jh\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.399188 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.398924 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-credential-socket\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.399188 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.398983 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-socket" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:45.399188 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399091 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-token\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-token\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.399188 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399153 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-data\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.399415 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399211 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-podinfo\") pod \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\" (UID: \"e481a487-1d1b-42f1-bc66-9638bceeb9c0\") " Oct 31 20:12:45.399415 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399213 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-certs" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:45.399415 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399292 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istiod-ca-cert" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:45.399584 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399439 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-data" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:45.399584 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399525 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-credential-socket" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:45.399813 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399793 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.399901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399815 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.399901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399825 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.399901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399842 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.399901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.399857 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.401446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.401422 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-envoy" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:12:45.401566 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.401540 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-kube-api-access-c24jh" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "kube-api-access-c24jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:45.401665 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.401648 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-token" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:45.401738 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.401723 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-podinfo" pod "e481a487-1d1b-42f1-bc66-9638bceeb9c0" (UID: "e481a487-1d1b-42f1-bc66-9638bceeb9c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:12:45.500976 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.500906 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.500976 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.500932 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24jh\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-kube-api-access-c24jh\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.500976 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.500944 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.500976 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.500954 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e481a487-1d1b-42f1-bc66-9638bceeb9c0-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:45.581512 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.581481 6481 generic.go:356] "Generic (PLEG): container finished" podID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerID="8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223" exitCode=0 Oct 31 20:12:45.581659 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.581528 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" Oct 31 20:12:45.581659 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.581568 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" event={"ID":"e481a487-1d1b-42f1-bc66-9638bceeb9c0","Type":"ContainerDied","Data":"8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223"} Oct 31 20:12:45.581659 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.581626 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd" event={"ID":"e481a487-1d1b-42f1-bc66-9638bceeb9c0","Type":"ContainerDied","Data":"5508f014db8d3da2800973e880707e4bc523fccc9abd8d1849ac6de8c496a237"} Oct 31 20:12:45.581659 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.581654 6481 scope.go:122] "RemoveContainer" containerID="8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223" Oct 31 20:12:45.591398 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.591384 6481 scope.go:122] "RemoveContainer" containerID="8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d" Oct 31 20:12:45.599974 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.599955 6481 scope.go:122] "RemoveContainer" containerID="8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223" Oct 31 20:12:45.600256 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:45.600237 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223\": container with ID starting with 8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223 not found: ID does not exist" containerID="8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223" Oct 31 20:12:45.600322 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.600269 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223"} err="failed to get container status \"8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223\": rpc error: code = NotFound desc = could not find container \"8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223\": container with ID starting with 8da35d591cb7e5370017a4aa0fbdf8a785836ee9c670312d4f06e55abbc47223 not found: ID does not exist" Oct 31 20:12:45.600322 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.600288 6481 scope.go:122] "RemoveContainer" containerID="8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d" Oct 31 20:12:45.600494 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:45.600480 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d\": container with ID starting with 8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d not found: ID does not exist" containerID="8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d" Oct 31 20:12:45.600536 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.600497 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d"} err="failed to get container status \"8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d\": rpc error: code = NotFound desc = could not find container \"8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d\": container with ID starting with 8b88ab4554fffe86dafa85a1453c1b8b1f19d3887600247695b8035b29e8077d not found: ID does not exist" Oct 31 20:12:45.748931 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.748895 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd"] Oct 31 20:12:45.752674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:45.752609 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5d59a2-openshift-default-68cdc5c4d5-mvbsd"] Oct 31 20:12:46.878949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:46.878904 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" path="/var/lib/kubelet/pods/e481a487-1d1b-42f1-bc66-9638bceeb9c0/volumes" Oct 31 20:12:57.325263 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:57.325176 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:12:58.242648 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.242614 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl"] Oct 31 20:12:58.242862 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.242835 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" podUID="08fbdeaa-939e-4700-bb30-a1a70c5da6bd" containerName="maas-api" containerID="cri-o://425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e" gracePeriod=30 Oct 31 20:12:58.534546 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.534519 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:12:58.659020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.657191 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-metrics-certs\") pod \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " Oct 31 20:12:58.659020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.657249 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-kube-api-access-x9h6j\" (UniqueName: \"kubernetes.io/projected/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-kube-api-access-x9h6j\") pod \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " Oct 31 20:12:58.659020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.657381 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-maas-api-tls\") pod \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " Oct 31 20:12:58.659020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.657471 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-openshift-service-ca\") pod \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\" (UID: \"08fbdeaa-939e-4700-bb30-a1a70c5da6bd\") " Oct 31 20:12:58.659020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.658443 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-openshift-service-ca" pod "08fbdeaa-939e-4700-bb30-a1a70c5da6bd" (UID: "08fbdeaa-939e-4700-bb30-a1a70c5da6bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:12:58.672985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.670947 6481 generic.go:356] "Generic (PLEG): container finished" podID="08fbdeaa-939e-4700-bb30-a1a70c5da6bd" containerID="425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e" exitCode=0 Oct 31 20:12:58.672985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.671013 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" event={"ID":"08fbdeaa-939e-4700-bb30-a1a70c5da6bd","Type":"ContainerDied","Data":"425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e"} Oct 31 20:12:58.672985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.671101 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" event={"ID":"08fbdeaa-939e-4700-bb30-a1a70c5da6bd","Type":"ContainerDied","Data":"01cd3933eb64874b505989f06953ffe20d7b754d879c703851e6a6ee311dff15"} Oct 31 20:12:58.672985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.671128 6481 scope.go:122] "RemoveContainer" containerID="425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e" Oct 31 20:12:58.672985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.671270 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl" Oct 31 20:12:58.684020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.682422 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-kube-api-access-x9h6j" pod "08fbdeaa-939e-4700-bb30-a1a70c5da6bd" (UID: "08fbdeaa-939e-4700-bb30-a1a70c5da6bd"). InnerVolumeSpecName "kube-api-access-x9h6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:12:58.684020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.683305 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-metrics-certs" pod "08fbdeaa-939e-4700-bb30-a1a70c5da6bd" (UID: "08fbdeaa-939e-4700-bb30-a1a70c5da6bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:58.692742 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.690683 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-maas-api-tls" pod "08fbdeaa-939e-4700-bb30-a1a70c5da6bd" (UID: "08fbdeaa-939e-4700-bb30-a1a70c5da6bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:12:58.707516 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.707486 6481 scope.go:122] "RemoveContainer" containerID="425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e" Oct 31 20:12:58.707901 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:12:58.707876 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e\": container with ID starting with 425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e not found: ID does not exist" containerID="425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e" Oct 31 20:12:58.708014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.707920 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e"} err="failed to get container status \"425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e\": rpc error: code = NotFound desc = could not find container \"425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e\": container with ID starting with 425d68e9048c73490c47f29ce31c4c3591be6162366f619a414b0bb484747f3e not found: ID does not exist" Oct 31 20:12:58.759358 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.759332 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:58.759358 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.759357 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9h6j\" (UniqueName: \"kubernetes.io/projected/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-kube-api-access-x9h6j\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:58.759522 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.759368 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:58.759522 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:58.759377 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fbdeaa-939e-4700-bb30-a1a70c5da6bd-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:12:59.089616 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:59.089585 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl"] Oct 31 20:12:59.091397 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:12:59.091377 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44dd-5cb7d98676-4q4sl"] Oct 31 20:13:00.878849 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:00.878807 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fbdeaa-939e-4700-bb30-a1a70c5da6bd" path="/var/lib/kubelet/pods/08fbdeaa-939e-4700-bb30-a1a70c5da6bd/volumes" Oct 31 20:13:06.921535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:06.921505 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96"] Oct 31 20:13:06.921928 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:06.921690 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" containerID="cri-o://25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30" gracePeriod=30 Oct 31 20:13:11.315766 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:11.315723 6481 patch_prober.go:29] interesting pod/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:13:11.316207 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:11.315787 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:13:12.171385 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.171355 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:13:12.328680 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328649 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cd6d98-800c-481c-9685-d737156993bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cd6d98-800c-481c-9685-d737156993bc-istiod-ca-cert\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328707 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-kube-api-access-tktqh\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-kube-api-access-tktqh\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328748 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-envoy\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328812 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-socket\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328841 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-credential-socket\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328867 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-data\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328892 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-certs\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.328930 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37cd6d98-800c-481c-9685-d737156993bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cd6d98-800c-481c-9685-d737156993bc-istio-podinfo\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329553 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329226 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-data" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:12.329553 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329239 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-certs" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:12.329553 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329273 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-credential-socket" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:12.329553 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329312 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-istio-token\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-istio-token\") pod \"37cd6d98-800c-481c-9685-d737156993bc\" (UID: \"37cd6d98-800c-481c-9685-d737156993bc\") " Oct 31 20:13:12.329553 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329412 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-socket" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:12.329819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329579 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cd6d98-800c-481c-9685-d737156993bc-istiod-ca-cert" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:12.329880 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329857 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.329932 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329876 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.329932 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329892 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.329932 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329906 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.329932 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.329919 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cd6d98-800c-481c-9685-d737156993bc-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.331065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.331037 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-envoy" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:12.331205 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.331162 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-kube-api-access-tktqh" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "kube-api-access-tktqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:12.331270 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.331252 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-istio-token" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:12.331509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.331483 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37cd6d98-800c-481c-9685-d737156993bc-istio-podinfo" pod "37cd6d98-800c-481c-9685-d737156993bc" (UID: "37cd6d98-800c-481c-9685-d737156993bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:13:12.430511 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.430480 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tktqh\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-kube-api-access-tktqh\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.430511 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.430510 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cd6d98-800c-481c-9685-d737156993bc-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.430740 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.430523 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cd6d98-800c-481c-9685-d737156993bc-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.430740 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.430535 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cd6d98-800c-481c-9685-d737156993bc-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:12.782343 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.782309 6481 generic.go:356] "Generic (PLEG): container finished" podID="37cd6d98-800c-481c-9685-d737156993bc" containerID="25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30" exitCode=0 Oct 31 20:13:12.782535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.782355 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" Oct 31 20:13:12.782535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.782388 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" event={"ID":"37cd6d98-800c-481c-9685-d737156993bc","Type":"ContainerDied","Data":"25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30"} Oct 31 20:13:12.782535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.782447 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96" event={"ID":"37cd6d98-800c-481c-9685-d737156993bc","Type":"ContainerDied","Data":"6f79f71747f81336052b9a91071189b27a5c6cdb15ba17c8286f4c79e45e03d1"} Oct 31 20:13:12.782535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.782477 6481 scope.go:122] "RemoveContainer" containerID="25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30" Oct 31 20:13:12.792067 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.792049 6481 scope.go:122] "RemoveContainer" containerID="9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c" Oct 31 20:13:12.800293 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.800278 6481 scope.go:122] "RemoveContainer" containerID="25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30" Oct 31 20:13:12.800529 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:12.800512 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30\": container with ID starting with 25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30 not found: ID does not exist" containerID="25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30" Oct 31 20:13:12.800595 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.800536 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30"} err="failed to get container status \"25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30\": rpc error: code = NotFound desc = could not find container \"25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30\": container with ID starting with 25fcabce12718710c6065074dde1ac99e68bc8d397c88282ad283828879ffe30 not found: ID does not exist" Oct 31 20:13:12.800595 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.800553 6481 scope.go:122] "RemoveContainer" containerID="9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c" Oct 31 20:13:12.800758 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:12.800744 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c\": container with ID starting with 9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c not found: ID does not exist" containerID="9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c" Oct 31 20:13:12.800798 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.800762 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c"} err="failed to get container status \"9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c\": rpc error: code = NotFound desc = could not find container \"9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c\": container with ID starting with 9aff7a6220ccf5d3e09733123884bd522944c552fafac26854c9d2f6a98b9b0c not found: ID does not exist" Oct 31 20:13:12.950261 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:12.950225 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96"] Oct 31 20:13:13.001056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:13.001025 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef44dd-openshift-default-86b756c5f9-ntl96"] Oct 31 20:13:14.878872 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:14.878836 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cd6d98-800c-481c-9685-d737156993bc" path="/var/lib/kubelet/pods/37cd6d98-800c-481c-9685-d737156993bc/volumes" Oct 31 20:13:25.752138 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:25.752105 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z"] Oct 31 20:13:25.752576 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:25.752320 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" podUID="618222d1-87d7-4407-ba25-77cc87fa12d4" containerName="maas-api" containerID="cri-o://9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca" gracePeriod=30 Oct 31 20:13:26.004453 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.004395 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 20:13:26.091546 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.091514 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-metrics-certs\") pod \"618222d1-87d7-4407-ba25-77cc87fa12d4\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " Oct 31 20:13:26.091797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.091600 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618222d1-87d7-4407-ba25-77cc87fa12d4-kube-api-access-xhkpn\" (UniqueName: \"kubernetes.io/projected/618222d1-87d7-4407-ba25-77cc87fa12d4-kube-api-access-xhkpn\") pod \"618222d1-87d7-4407-ba25-77cc87fa12d4\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " Oct 31 20:13:26.091797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.091673 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618222d1-87d7-4407-ba25-77cc87fa12d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618222d1-87d7-4407-ba25-77cc87fa12d4-openshift-service-ca\") pod \"618222d1-87d7-4407-ba25-77cc87fa12d4\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " Oct 31 20:13:26.091797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.091774 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls\") pod \"618222d1-87d7-4407-ba25-77cc87fa12d4\" (UID: \"618222d1-87d7-4407-ba25-77cc87fa12d4\") " Oct 31 20:13:26.092058 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.092034 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618222d1-87d7-4407-ba25-77cc87fa12d4-openshift-service-ca" pod "618222d1-87d7-4407-ba25-77cc87fa12d4" (UID: "618222d1-87d7-4407-ba25-77cc87fa12d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:26.092188 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.092171 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618222d1-87d7-4407-ba25-77cc87fa12d4-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:26.093671 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.093652 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-metrics-certs" pod "618222d1-87d7-4407-ba25-77cc87fa12d4" (UID: "618222d1-87d7-4407-ba25-77cc87fa12d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:26.093746 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.093711 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618222d1-87d7-4407-ba25-77cc87fa12d4-kube-api-access-xhkpn" pod "618222d1-87d7-4407-ba25-77cc87fa12d4" (UID: "618222d1-87d7-4407-ba25-77cc87fa12d4"). InnerVolumeSpecName "kube-api-access-xhkpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:26.093785 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.093766 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls" pod "618222d1-87d7-4407-ba25-77cc87fa12d4" (UID: "618222d1-87d7-4407-ba25-77cc87fa12d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:26.192765 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.192732 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:26.192765 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.192757 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618222d1-87d7-4407-ba25-77cc87fa12d4-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:26.192765 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.192768 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhkpn\" (UniqueName: \"kubernetes.io/projected/618222d1-87d7-4407-ba25-77cc87fa12d4-kube-api-access-xhkpn\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:26.875403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.875369 6481 generic.go:356] "Generic (PLEG): container finished" podID="618222d1-87d7-4407-ba25-77cc87fa12d4" containerID="9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca" exitCode=0 Oct 31 20:13:26.875779 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.875494 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" Oct 31 20:13:26.878668 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.878630 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" event={"ID":"618222d1-87d7-4407-ba25-77cc87fa12d4","Type":"ContainerDied","Data":"9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca"} Oct 31 20:13:26.878861 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.878693 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z" event={"ID":"618222d1-87d7-4407-ba25-77cc87fa12d4","Type":"ContainerDied","Data":"189ba34e5d67e71d1899d7b62c0b98dd5babbc33218ac30fa86386cb3c8f43ed"} Oct 31 20:13:26.878861 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.878740 6481 scope.go:122] "RemoveContainer" containerID="9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca" Oct 31 20:13:26.888897 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.888876 6481 scope.go:122] "RemoveContainer" containerID="9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca" Oct 31 20:13:26.889191 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:26.889170 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca\": container with ID starting with 9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca not found: ID does not exist" containerID="9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca" Oct 31 20:13:26.889246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:26.889200 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca"} err="failed to get container status \"9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca\": rpc error: code = NotFound desc = could not find container \"9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca\": container with ID starting with 9add5a78e60f47e965f2681f83b811b78d93c0e8a487d76af9bac4f8f777c6ca not found: ID does not exist" Oct 31 20:13:27.043177 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:27.043146 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z"] Oct 31 20:13:27.094228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:27.094195 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-872fee-6d85cf4f9f-t548z"] Oct 31 20:13:27.323891 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:27.323858 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:13:28.879019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:28.878964 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618222d1-87d7-4407-ba25-77cc87fa12d4" path="/var/lib/kubelet/pods/618222d1-87d7-4407-ba25-77cc87fa12d4/volumes" Oct 31 20:13:32.081430 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.081393 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j"] Oct 31 20:13:32.081944 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.081685 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" podUID="82621e2b-ff5a-45b2-b468-5565a05f7f0a" containerName="maas-api" containerID="cri-o://a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357" gracePeriod=30 Oct 31 20:13:32.348403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.348380 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 20:13:32.476572 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.475394 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-metrics-certs\") pod \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " Oct 31 20:13:32.476572 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.475454 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82621e2b-ff5a-45b2-b468-5565a05f7f0a-kube-api-access-prqtr\" (UniqueName: \"kubernetes.io/projected/82621e2b-ff5a-45b2-b468-5565a05f7f0a-kube-api-access-prqtr\") pod \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " Oct 31 20:13:32.476572 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.475716 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls\") pod \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " Oct 31 20:13:32.476572 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.475819 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82621e2b-ff5a-45b2-b468-5565a05f7f0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82621e2b-ff5a-45b2-b468-5565a05f7f0a-openshift-service-ca\") pod \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\" (UID: \"82621e2b-ff5a-45b2-b468-5565a05f7f0a\") " Oct 31 20:13:32.477247 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.477218 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82621e2b-ff5a-45b2-b468-5565a05f7f0a-openshift-service-ca" pod "82621e2b-ff5a-45b2-b468-5565a05f7f0a" (UID: "82621e2b-ff5a-45b2-b468-5565a05f7f0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:32.493610 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.493359 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82621e2b-ff5a-45b2-b468-5565a05f7f0a-kube-api-access-prqtr" pod "82621e2b-ff5a-45b2-b468-5565a05f7f0a" (UID: "82621e2b-ff5a-45b2-b468-5565a05f7f0a"). InnerVolumeSpecName "kube-api-access-prqtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:32.512047 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.503979 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls" pod "82621e2b-ff5a-45b2-b468-5565a05f7f0a" (UID: "82621e2b-ff5a-45b2-b468-5565a05f7f0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:32.512542 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.512507 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-metrics-certs" pod "82621e2b-ff5a-45b2-b468-5565a05f7f0a" (UID: "82621e2b-ff5a-45b2-b468-5565a05f7f0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:32.593821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.590412 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:32.593821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.593661 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82621e2b-ff5a-45b2-b468-5565a05f7f0a-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:32.593821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.593693 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82621e2b-ff5a-45b2-b468-5565a05f7f0a-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:32.593821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.593708 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prqtr\" (UniqueName: \"kubernetes.io/projected/82621e2b-ff5a-45b2-b468-5565a05f7f0a-kube-api-access-prqtr\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:32.914092 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.914061 6481 generic.go:356] "Generic (PLEG): container finished" podID="82621e2b-ff5a-45b2-b468-5565a05f7f0a" containerID="a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357" exitCode=0 Oct 31 20:13:32.914250 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.914108 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" event={"ID":"82621e2b-ff5a-45b2-b468-5565a05f7f0a","Type":"ContainerDied","Data":"a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357"} Oct 31 20:13:32.914250 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.914112 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" Oct 31 20:13:32.914250 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.914145 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j" event={"ID":"82621e2b-ff5a-45b2-b468-5565a05f7f0a","Type":"ContainerDied","Data":"efb26071488741532830d1738128d1d6dfb061fda7d38dca2522eab23d6b4f01"} Oct 31 20:13:32.914250 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.914165 6481 scope.go:122] "RemoveContainer" containerID="a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357" Oct 31 20:13:32.922946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.922925 6481 scope.go:122] "RemoveContainer" containerID="a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357" Oct 31 20:13:32.923221 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:32.923203 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357\": container with ID starting with a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357 not found: ID does not exist" containerID="a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357" Oct 31 20:13:32.923279 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:32.923229 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357"} err="failed to get container status \"a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357\": rpc error: code = NotFound desc = could not find container \"a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357\": container with ID starting with a9dda98e9a847751f46f2f69be7d549565e9da72e17921ec42143fc47ae92357 not found: ID does not exist" Oct 31 20:13:33.087097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:33.087059 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j"] Oct 31 20:13:33.088192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:33.088169 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-967dee-bb58ff99c-6bb6j"] Oct 31 20:13:34.878884 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:34.878852 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82621e2b-ff5a-45b2-b468-5565a05f7f0a" path="/var/lib/kubelet/pods/82621e2b-ff5a-45b2-b468-5565a05f7f0a/volumes" Oct 31 20:13:35.098837 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:35.098806 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr"] Oct 31 20:13:35.099056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:35.099026 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" containerID="cri-o://a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c" gracePeriod=30 Oct 31 20:13:36.359558 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:36.359519 6481 patch_prober.go:29] interesting pod/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:13:36.359949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:36.359589 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:13:40.352706 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.352676 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 20:13:40.479258 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479227 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-token\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-token\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479311 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-podinfo\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479350 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-socket\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479369 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cabd44-169b-40d1-82a5-eb3588801cbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cabd44-169b-40d1-82a5-eb3588801cbe-istiod-ca-cert\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479387 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-envoy\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479416 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-kube-api-access-9d4jv\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-kube-api-access-9d4jv\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479703 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479519 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-certs\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479703 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479607 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-credential-socket\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479888 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479720 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-data\") pod \"98cabd44-169b-40d1-82a5-eb3588801cbe\" (UID: \"98cabd44-169b-40d1-82a5-eb3588801cbe\") " Oct 31 20:13:40.479888 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479719 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-socket" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:40.479888 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479822 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-certs" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:40.479888 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479856 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cabd44-169b-40d1-82a5-eb3588801cbe-istiod-ca-cert" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:40.480084 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.479946 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-credential-socket" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:40.480154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.480104 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-data" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:40.480320 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.480293 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.480446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.480322 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cabd44-169b-40d1-82a5-eb3588801cbe-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.480446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.480337 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.480446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.480350 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.480446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.480364 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.481595 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.481572 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-token" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:40.481863 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.481839 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-podinfo" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:13:40.481977 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.481958 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-envoy" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:13:40.482273 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.481976 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-kube-api-access-9d4jv" pod "98cabd44-169b-40d1-82a5-eb3588801cbe" (UID: "98cabd44-169b-40d1-82a5-eb3588801cbe"). InnerVolumeSpecName "kube-api-access-9d4jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:40.581335 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.581304 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.581335 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.581334 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.581528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.581346 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cabd44-169b-40d1-82a5-eb3588801cbe-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.581528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.581361 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4jv\" (UniqueName: \"kubernetes.io/projected/98cabd44-169b-40d1-82a5-eb3588801cbe-kube-api-access-9d4jv\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:40.958243 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.958208 6481 generic.go:356] "Generic (PLEG): container finished" podID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerID="a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c" exitCode=0 Oct 31 20:13:40.958403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.958281 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" Oct 31 20:13:40.958403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.958378 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" event={"ID":"98cabd44-169b-40d1-82a5-eb3588801cbe","Type":"ContainerDied","Data":"a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c"} Oct 31 20:13:40.958536 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.958437 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr" event={"ID":"98cabd44-169b-40d1-82a5-eb3588801cbe","Type":"ContainerDied","Data":"7d11917ef462b8c34eba34d8ec7ddcbcfb3b87b5161bcd49d6fce4a1e49d5aa2"} Oct 31 20:13:40.958536 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.958465 6481 scope.go:122] "RemoveContainer" containerID="a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c" Oct 31 20:13:40.969560 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.969542 6481 scope.go:122] "RemoveContainer" containerID="146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e" Oct 31 20:13:40.977758 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.977735 6481 scope.go:122] "RemoveContainer" containerID="a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c" Oct 31 20:13:40.978058 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:40.978035 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c\": container with ID starting with a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c not found: ID does not exist" containerID="a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c" Oct 31 20:13:40.978153 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.978071 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c"} err="failed to get container status \"a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c\": rpc error: code = NotFound desc = could not find container \"a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c\": container with ID starting with a940d15d3d96b2b8b554dee1500a094c8d54add28289c902f7f6f68b55cd9b7c not found: ID does not exist" Oct 31 20:13:40.978153 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.978090 6481 scope.go:122] "RemoveContainer" containerID="146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e" Oct 31 20:13:40.978346 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:40.978325 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e\": container with ID starting with 146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e not found: ID does not exist" containerID="146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e" Oct 31 20:13:40.978387 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:40.978352 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e"} err="failed to get container status \"146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e\": rpc error: code = NotFound desc = could not find container \"146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e\": container with ID starting with 146061569b3ceb9b6da6f0e68daee4774d5bec554158a3c2dc3a6c27f3ea830e not found: ID does not exist" Oct 31 20:13:41.128577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:41.128546 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr"] Oct 31 20:13:41.133195 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:41.133170 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-872fee-openshift-default-7b9695b88f-x8phr"] Oct 31 20:13:42.878426 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:42.878390 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" path="/var/lib/kubelet/pods/98cabd44-169b-40d1-82a5-eb3588801cbe/volumes" Oct 31 20:13:47.373646 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:47.373611 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v"] Oct 31 20:13:47.480627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:47.480591 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv"] Oct 31 20:13:52.373884 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.373833 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" podUID="b24f1601-6686-49d2-b525-5b302a70c2cf" containerName="payload-processing" containerID="cri-o://eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12" gracePeriod=25 Oct 31 20:13:52.481959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.481896 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" podUID="3def61aa-6d85-4e51-88fa-cf4866d4a9f0" containerName="payload-pre-processing" containerID="cri-o://07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47" gracePeriod=25 Oct 31 20:13:52.641813 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.641748 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 20:13:52.712883 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.712853 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24f1601-6686-49d2-b525-5b302a70c2cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24f1601-6686-49d2-b525-5b302a70c2cf-plugins-config-volume\") pod \"b24f1601-6686-49d2-b525-5b302a70c2cf\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " Oct 31 20:13:52.713076 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.712982 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24f1601-6686-49d2-b525-5b302a70c2cf-kube-api-access-v4psj\" (UniqueName: \"kubernetes.io/projected/b24f1601-6686-49d2-b525-5b302a70c2cf-kube-api-access-v4psj\") pod \"b24f1601-6686-49d2-b525-5b302a70c2cf\" (UID: \"b24f1601-6686-49d2-b525-5b302a70c2cf\") " Oct 31 20:13:52.713304 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.713280 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24f1601-6686-49d2-b525-5b302a70c2cf-plugins-config-volume" pod "b24f1601-6686-49d2-b525-5b302a70c2cf" (UID: "b24f1601-6686-49d2-b525-5b302a70c2cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:52.713378 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.713366 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24f1601-6686-49d2-b525-5b302a70c2cf-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:52.715743 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.715719 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24f1601-6686-49d2-b525-5b302a70c2cf-kube-api-access-v4psj" pod "b24f1601-6686-49d2-b525-5b302a70c2cf" (UID: "b24f1601-6686-49d2-b525-5b302a70c2cf"). InnerVolumeSpecName "kube-api-access-v4psj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:52.736478 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.736456 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 20:13:52.814271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.814238 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-plugins-config-volume\") pod \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " Oct 31 20:13:52.814483 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.814325 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-kube-api-access-9hc54\" (UniqueName: \"kubernetes.io/projected/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-kube-api-access-9hc54\") pod \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\" (UID: \"3def61aa-6d85-4e51-88fa-cf4866d4a9f0\") " Oct 31 20:13:52.814559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.814530 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4psj\" (UniqueName: \"kubernetes.io/projected/b24f1601-6686-49d2-b525-5b302a70c2cf-kube-api-access-v4psj\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:52.814720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.814678 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-plugins-config-volume" pod "3def61aa-6d85-4e51-88fa-cf4866d4a9f0" (UID: "3def61aa-6d85-4e51-88fa-cf4866d4a9f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:52.816394 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.816375 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-kube-api-access-9hc54" pod "3def61aa-6d85-4e51-88fa-cf4866d4a9f0" (UID: "3def61aa-6d85-4e51-88fa-cf4866d4a9f0"). InnerVolumeSpecName "kube-api-access-9hc54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:52.915421 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.915336 6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-plugins-config-volume\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:52.915421 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:52.915366 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hc54\" (UniqueName: \"kubernetes.io/projected/3def61aa-6d85-4e51-88fa-cf4866d4a9f0-kube-api-access-9hc54\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:53.041532 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.041497 6481 generic.go:356] "Generic (PLEG): container finished" podID="b24f1601-6686-49d2-b525-5b302a70c2cf" containerID="eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12" exitCode=0 Oct 31 20:13:53.041727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.041542 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" Oct 31 20:13:53.041727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.041584 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" event={"ID":"b24f1601-6686-49d2-b525-5b302a70c2cf","Type":"ContainerDied","Data":"eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12"} Oct 31 20:13:53.041727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.041640 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v" event={"ID":"b24f1601-6686-49d2-b525-5b302a70c2cf","Type":"ContainerDied","Data":"ebe5f0147150290bbd521acacc97424ae6b5ae59e0ddd5ae86b6459ba9fca3aa"} Oct 31 20:13:53.041727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.041655 6481 scope.go:122] "RemoveContainer" containerID="eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12" Oct 31 20:13:53.043277 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.043252 6481 generic.go:356] "Generic (PLEG): container finished" podID="3def61aa-6d85-4e51-88fa-cf4866d4a9f0" containerID="07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47" exitCode=0 Oct 31 20:13:53.043390 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.043287 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" event={"ID":"3def61aa-6d85-4e51-88fa-cf4866d4a9f0","Type":"ContainerDied","Data":"07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47"} Oct 31 20:13:53.043390 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.043323 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" event={"ID":"3def61aa-6d85-4e51-88fa-cf4866d4a9f0","Type":"ContainerDied","Data":"83f4604f23f12bafb4932e50b7fe578769a1b22ccdc3dd04294e1c874c5cf4cc"} Oct 31 20:13:53.043390 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.043327 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv" Oct 31 20:13:53.051019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.050985 6481 scope.go:122] "RemoveContainer" containerID="eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12" Oct 31 20:13:53.051311 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:53.051290 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12\": container with ID starting with eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12 not found: ID does not exist" containerID="eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12" Oct 31 20:13:53.051365 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.051321 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12"} err="failed to get container status \"eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12\": rpc error: code = NotFound desc = could not find container \"eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12\": container with ID starting with eb079b2b53f73dd6b22dbc7e300cfe09e6cbfb8ae66b2ed25349ffdda2cf5a12 not found: ID does not exist" Oct 31 20:13:53.051365 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.051339 6481 scope.go:122] "RemoveContainer" containerID="07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47" Oct 31 20:13:53.059312 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.059291 6481 scope.go:122] "RemoveContainer" containerID="07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47" Oct 31 20:13:53.059571 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:53.059543 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47\": container with ID starting with 07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47 not found: ID does not exist" containerID="07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47" Oct 31 20:13:53.059625 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.059570 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47"} err="failed to get container status \"07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47\": rpc error: code = NotFound desc = could not find container \"07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47\": container with ID starting with 07f6cf05f9e1fcf835fc0b7e07318415de847586156c00c237402f762dc11c47 not found: ID does not exist" Oct 31 20:13:53.208917 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.208835 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv"] Oct 31 20:13:53.214099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.214061 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6eeb83-54f4d447d7-dlzdv"] Oct 31 20:13:53.325664 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.325632 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v"] Oct 31 20:13:53.327842 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:53.327823 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6eeb83-776cfcc85-7k79v"] Oct 31 20:13:54.878910 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:54.878881 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3def61aa-6d85-4e51-88fa-cf4866d4a9f0" path="/var/lib/kubelet/pods/3def61aa-6d85-4e51-88fa-cf4866d4a9f0/volumes" Oct 31 20:13:54.879390 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:54.879373 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24f1601-6686-49d2-b525-5b302a70c2cf" path="/var/lib/kubelet/pods/b24f1601-6686-49d2-b525-5b302a70c2cf/volumes" Oct 31 20:13:55.088634 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:55.088600 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2"] Oct 31 20:13:55.088808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:55.088788 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" containerID="cri-o://1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc" gracePeriod=30 Oct 31 20:13:55.140977 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:55.140910 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s"] Oct 31 20:13:55.141206 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:55.141178 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" containerID="cri-o://f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a" gracePeriod=30 Oct 31 20:13:57.325128 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:57.325091 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:13:58.277732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.277695 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs"] Oct 31 20:13:58.277959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.277933 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" podUID="9da63ab8-4e17-473f-b41d-879a1e89350c" containerName="maas-api" containerID="cri-o://1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c" gracePeriod=30 Oct 31 20:13:58.547033 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.546990 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 20:13:58.570553 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.570518 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da63ab8-4e17-473f-b41d-879a1e89350c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da63ab8-4e17-473f-b41d-879a1e89350c-openshift-service-ca\") pod \"9da63ab8-4e17-473f-b41d-879a1e89350c\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " Oct 31 20:13:58.570719 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.570634 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-metrics-certs\") pod \"9da63ab8-4e17-473f-b41d-879a1e89350c\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " Oct 31 20:13:58.570719 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.570675 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da63ab8-4e17-473f-b41d-879a1e89350c-kube-api-access-qb84c\" (UniqueName: \"kubernetes.io/projected/9da63ab8-4e17-473f-b41d-879a1e89350c-kube-api-access-qb84c\") pod \"9da63ab8-4e17-473f-b41d-879a1e89350c\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " Oct 31 20:13:58.570828 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.570806 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls\") pod \"9da63ab8-4e17-473f-b41d-879a1e89350c\" (UID: \"9da63ab8-4e17-473f-b41d-879a1e89350c\") " Oct 31 20:13:58.570944 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.570924 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da63ab8-4e17-473f-b41d-879a1e89350c-openshift-service-ca" pod "9da63ab8-4e17-473f-b41d-879a1e89350c" (UID: "9da63ab8-4e17-473f-b41d-879a1e89350c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:13:58.571174 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.571156 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da63ab8-4e17-473f-b41d-879a1e89350c-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:58.572666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.572642 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-metrics-certs" pod "9da63ab8-4e17-473f-b41d-879a1e89350c" (UID: "9da63ab8-4e17-473f-b41d-879a1e89350c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:58.572762 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.572712 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da63ab8-4e17-473f-b41d-879a1e89350c-kube-api-access-qb84c" pod "9da63ab8-4e17-473f-b41d-879a1e89350c" (UID: "9da63ab8-4e17-473f-b41d-879a1e89350c"). InnerVolumeSpecName "kube-api-access-qb84c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:13:58.572960 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.572938 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls" pod "9da63ab8-4e17-473f-b41d-879a1e89350c" (UID: "9da63ab8-4e17-473f-b41d-879a1e89350c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:13:58.689763 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.689678 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:58.689763 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.689719 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da63ab8-4e17-473f-b41d-879a1e89350c-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:58.689763 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:58.689734 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb84c\" (UniqueName: \"kubernetes.io/projected/9da63ab8-4e17-473f-b41d-879a1e89350c-kube-api-access-qb84c\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:13:59.100989 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.100953 6481 generic.go:356] "Generic (PLEG): container finished" podID="9da63ab8-4e17-473f-b41d-879a1e89350c" containerID="1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c" exitCode=0 Oct 31 20:13:59.101164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.101005 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" Oct 31 20:13:59.101164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.101008 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" event={"ID":"9da63ab8-4e17-473f-b41d-879a1e89350c","Type":"ContainerDied","Data":"1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c"} Oct 31 20:13:59.101164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.101115 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs" event={"ID":"9da63ab8-4e17-473f-b41d-879a1e89350c","Type":"ContainerDied","Data":"0757074ab628c022155b0f920a43dda1483dcad89c5bebdd87610940a37f94fa"} Oct 31 20:13:59.101164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.101139 6481 scope.go:122] "RemoveContainer" containerID="1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c" Oct 31 20:13:59.109934 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.109916 6481 scope.go:122] "RemoveContainer" containerID="1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c" Oct 31 20:13:59.110177 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:13:59.110160 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c\": container with ID starting with 1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c not found: ID does not exist" containerID="1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c" Oct 31 20:13:59.110227 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.110187 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c"} err="failed to get container status \"1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c\": rpc error: code = NotFound desc = could not find container \"1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c\": container with ID starting with 1343ade26721cd3d62682ebd563062ff9c6d15de11b0c2f707dbcfa62a53425c not found: ID does not exist" Oct 31 20:13:59.269911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.269880 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs"] Oct 31 20:13:59.272920 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:13:59.272890 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6eeb83-fc74d9686-ftdhs"] Oct 31 20:14:00.418207 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.418182 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:14:00.436966 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.436943 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:14:00.512323 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512289 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-kube-api-access-f9vbd\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-kube-api-access-f9vbd\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.512323 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512325 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istiod-ca-cert\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.512538 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512362 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-certs\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.512538 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512474 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-envoy\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.512627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512557 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-kube-api-access-tc4nt\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-kube-api-access-tc4nt\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.512627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512598 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-certs\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.512732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512636 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-envoy\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.512732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512621 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-certs" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.512732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512682 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-socket\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.512732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512726 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-socket\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.512967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512756 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-credential-socket\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.512967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512755 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istiod-ca-cert" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:00.512967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512788 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-podinfo\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.512967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512816 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-data\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.512967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512843 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-data\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.512967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512921 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-credential-socket\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.512967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512955 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-podinfo\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.513394 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.512986 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istiod-ca-cert\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.513394 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513049 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-token\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-token\") pod \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\" (UID: \"0f278bd1-bf4f-4932-a3bd-7f02d90b6173\") " Oct 31 20:14:00.513394 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513073 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-socket" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.513394 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513081 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-token\") pod \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\" (UID: \"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7\") " Oct 31 20:14:00.513394 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513218 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-certs" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.513652 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513451 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-socket" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.513652 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513576 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-credential-socket" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.513652 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513615 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-credential-socket" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.513832 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513787 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-data" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.513977 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.513953 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-data" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.514094 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514055 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istiod-ca-cert" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514360 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514387 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514402 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514416 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514429 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514442 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514454 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514466 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514480 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.514578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.514494 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.515267 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.515165 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-kube-api-access-f9vbd" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "kube-api-access-f9vbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:00.515330 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.515299 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-token" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:00.515604 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.515577 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-envoy" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.515749 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.515632 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-envoy" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:00.515749 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.515649 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-kube-api-access-tc4nt" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "kube-api-access-tc4nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:00.515943 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.515921 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-podinfo" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:14:00.516255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.516235 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-token" pod "0f278bd1-bf4f-4932-a3bd-7f02d90b6173" (UID: "0f278bd1-bf4f-4932-a3bd-7f02d90b6173"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:00.516849 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.516832 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-podinfo" pod "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" (UID: "c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:14:00.616286 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616243 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4nt\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-kube-api-access-tc4nt\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.616286 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616285 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.616722 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616299 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.616722 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616311 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.616722 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616510 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.616722 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616536 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.616722 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616562 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vbd\" (UniqueName: \"kubernetes.io/projected/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7-kube-api-access-f9vbd\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.616722 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.616576 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f278bd1-bf4f-4932-a3bd-7f02d90b6173-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:00.878072 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:00.878044 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da63ab8-4e17-473f-b41d-879a1e89350c" path="/var/lib/kubelet/pods/9da63ab8-4e17-473f-b41d-879a1e89350c/volumes" Oct 31 20:14:01.113876 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.113839 6481 generic.go:356] "Generic (PLEG): container finished" podID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerID="f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a" exitCode=0 Oct 31 20:14:01.114061 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.113885 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" Oct 31 20:14:01.114061 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.113920 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" event={"ID":"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7","Type":"ContainerDied","Data":"f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a"} Oct 31 20:14:01.114061 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.113988 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s" event={"ID":"c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7","Type":"ContainerDied","Data":"e466e384cadfac56d404a05f6134fe1472aabeb07ba7f5c513bd8f70d9c79772"} Oct 31 20:14:01.114061 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.114032 6481 scope.go:122] "RemoveContainer" containerID="f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a" Oct 31 20:14:01.124754 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.124718 6481 generic.go:356] "Generic (PLEG): container finished" podID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerID="1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc" exitCode=0 Oct 31 20:14:01.124882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.124780 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" event={"ID":"0f278bd1-bf4f-4932-a3bd-7f02d90b6173","Type":"ContainerDied","Data":"1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc"} Oct 31 20:14:01.124882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.124851 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" event={"ID":"0f278bd1-bf4f-4932-a3bd-7f02d90b6173","Type":"ContainerDied","Data":"28838c6f9cd55ce44ab929c005d88ce864bd54c3b15ff51e9856ebe39471ab46"} Oct 31 20:14:01.124882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.124869 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2" Oct 31 20:14:01.132372 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.132352 6481 scope.go:122] "RemoveContainer" containerID="f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a" Oct 31 20:14:01.132632 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:14:01.132615 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a\": container with ID starting with f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a not found: ID does not exist" containerID="f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a" Oct 31 20:14:01.132713 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.132638 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a"} err="failed to get container status \"f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a\": rpc error: code = NotFound desc = could not find container \"f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a\": container with ID starting with f56f913b5ddd2def785ea2c7b0c504bfcaac6d999d7e60543b790de96106183a not found: ID does not exist" Oct 31 20:14:01.132713 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.132653 6481 scope.go:122] "RemoveContainer" containerID="1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc" Oct 31 20:14:01.140718 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.140696 6481 scope.go:122] "RemoveContainer" containerID="1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc" Oct 31 20:14:01.140976 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:14:01.140957 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc\": container with ID starting with 1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc not found: ID does not exist" containerID="1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc" Oct 31 20:14:01.141062 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.140987 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc"} err="failed to get container status \"1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc\": rpc error: code = NotFound desc = could not find container \"1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc\": container with ID starting with 1d563295350b6b18fc5426b7a7eb94d329415906dd9a5c78ca452375461f57bc not found: ID does not exist" Oct 31 20:14:01.280863 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.280832 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2"] Oct 31 20:14:01.284732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.284709 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vpzf2"] Oct 31 20:14:01.398537 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.398462 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s"] Oct 31 20:14:01.402104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:01.402084 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rj66s"] Oct 31 20:14:02.878119 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:02.878089 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" path="/var/lib/kubelet/pods/0f278bd1-bf4f-4932-a3bd-7f02d90b6173/volumes" Oct 31 20:14:02.878544 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:02.878531 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" path="/var/lib/kubelet/pods/c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7/volumes" Oct 31 20:14:07.008827 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.008793 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7"] Oct 31 20:14:07.009248 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.009078 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" containerID="cri-o://21a0f438b96955476b0867592b092fd13bab50cc82c55c1a7bdea926e68e685a" gracePeriod=30 Oct 31 20:14:07.985209 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985176 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd"] Oct 31 20:14:07.985566 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985553 6481 cpu_manager.go:414] "containerMap: removing container" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" Oct 31 20:14:07.985613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985566 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" Oct 31 20:14:07.985613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985577 6481 cpu_manager.go:414] "containerMap: removing container" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" Oct 31 20:14:07.985613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985582 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" Oct 31 20:14:07.985613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985594 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" Oct 31 20:14:07.985613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985598 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" Oct 31 20:14:07.985613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985609 6481 cpu_manager.go:414] "containerMap: removing container" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" Oct 31 20:14:07.985613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985613 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985620 6481 cpu_manager.go:414] "containerMap: removing container" podUID="2ed78df0-07da-4e7e-be0c-964f5715ff45" containerName="main" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985625 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed78df0-07da-4e7e-be0c-964f5715ff45" containerName="main" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985637 6481 cpu_manager.go:414] "containerMap: removing container" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985641 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985648 6481 cpu_manager.go:414] "containerMap: removing container" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985653 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985659 6481 cpu_manager.go:414] "containerMap: removing container" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985663 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985669 6481 cpu_manager.go:414] "containerMap: removing container" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985673 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985678 6481 cpu_manager.go:414] "containerMap: removing container" podUID="b8249072-4b53-4698-93f2-3d0e192e20e7" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985682 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8249072-4b53-4698-93f2-3d0e192e20e7" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985687 6481 cpu_manager.go:414] "containerMap: removing container" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985691 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985700 6481 cpu_manager.go:414] "containerMap: removing container" podUID="08fbdeaa-939e-4700-bb30-a1a70c5da6bd" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985704 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fbdeaa-939e-4700-bb30-a1a70c5da6bd" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985710 6481 cpu_manager.go:414] "containerMap: removing container" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985714 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985721 6481 cpu_manager.go:414] "containerMap: removing container" podUID="618222d1-87d7-4407-ba25-77cc87fa12d4" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985725 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618222d1-87d7-4407-ba25-77cc87fa12d4" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985735 6481 cpu_manager.go:414] "containerMap: removing container" podUID="7eba519c-3954-4e63-ac58-384d264c1494" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985739 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eba519c-3954-4e63-ac58-384d264c1494" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985745 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985749 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985755 6481 cpu_manager.go:414] "containerMap: removing container" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985759 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985764 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3def61aa-6d85-4e51-88fa-cf4866d4a9f0" containerName="payload-pre-processing" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985769 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3def61aa-6d85-4e51-88fa-cf4866d4a9f0" containerName="payload-pre-processing" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985777 6481 cpu_manager.go:414] "containerMap: removing container" podUID="82621e2b-ff5a-45b2-b468-5565a05f7f0a" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985781 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82621e2b-ff5a-45b2-b468-5565a05f7f0a" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985791 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ece716af-28ae-4bb2-a370-e20910206680" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985797 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece716af-28ae-4bb2-a370-e20910206680" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985804 6481 cpu_manager.go:414] "containerMap: removing container" podUID="49713925-9c71-440b-94d3-79818cd211f8" containerName="main" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985809 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49713925-9c71-440b-94d3-79818cd211f8" containerName="main" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985815 6481 cpu_manager.go:414] "containerMap: removing container" podUID="b24f1601-6686-49d2-b525-5b302a70c2cf" containerName="payload-processing" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985819 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f1601-6686-49d2-b525-5b302a70c2cf" containerName="payload-processing" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985825 6481 cpu_manager.go:414] "containerMap: removing container" podUID="9da63ab8-4e17-473f-b41d-879a1e89350c" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985829 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da63ab8-4e17-473f-b41d-879a1e89350c" containerName="maas-api" Oct 31 20:14:07.985817 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985835 6481 cpu_manager.go:414] "containerMap: removing container" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985840 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985845 6481 cpu_manager.go:414] "containerMap: removing container" podUID="4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985849 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985902 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d8e539-fd47-4c03-8ae3-90ddeb4ad7c7" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985910 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c67c4b-3638-4895-b372-08d232d4ce2f" containerName="main" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985916 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed78df0-07da-4e7e-be0c-964f5715ff45" containerName="main" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985923 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f278bd1-bf4f-4932-a3bd-7f02d90b6173" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985929 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e18125-290b-4f97-acd8-0ab00d15f0db" containerName="main" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985933 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa138a5-163c-48aa-9254-02e50743b0b8" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985939 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985943 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8249072-4b53-4698-93f2-3d0e192e20e7" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985948 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985953 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fbdeaa-939e-4700-bb30-a1a70c5da6bd" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985957 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cd6d98-800c-481c-9685-d737156993bc" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985962 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="618222d1-87d7-4407-ba25-77cc87fa12d4" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985969 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eba519c-3954-4e63-ac58-384d264c1494" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985973 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfd8c51-9f8c-454e-89ec-c7f61dbc0d57" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985977 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cabd44-169b-40d1-82a5-eb3588801cbe" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985981 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3def61aa-6d85-4e51-88fa-cf4866d4a9f0" containerName="payload-pre-processing" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.985985 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="82621e2b-ff5a-45b2-b468-5565a05f7f0a" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986016 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece716af-28ae-4bb2-a370-e20910206680" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986020 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="49713925-9c71-440b-94d3-79818cd211f8" containerName="main" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986025 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f1601-6686-49d2-b525-5b302a70c2cf" containerName="payload-processing" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986030 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da63ab8-4e17-473f-b41d-879a1e89350c" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986034 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6ca183-e38a-43ba-bd2e-8d444dbcc29a" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986039 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb91e5e-1bf7-4449-9c55-c1fb2a6211e3" containerName="maas-api" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986046 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" Oct 31 20:14:07.986946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.986050 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e481a487-1d1b-42f1-bc66-9638bceeb9c0" containerName="istio-proxy" Oct 31 20:14:07.991222 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:07.991204 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.049644 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.049612 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd"] Oct 31 20:14:08.098193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.098158 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:08.098371 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.098276 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:14:08.100904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.100878 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzr8\" (UniqueName: \"kubernetes.io/projected/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-kube-api-access-dnzr8\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.100956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.100919 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.100987 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.100951 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.202205 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.202175 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzr8\" (UniqueName: \"kubernetes.io/projected/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-kube-api-access-dnzr8\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.202205 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.202209 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.202439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.202231 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.202972 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.202951 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.204634 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.204611 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.260577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.260514 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzr8\" (UniqueName: \"kubernetes.io/projected/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-kube-api-access-dnzr8\") pod \"maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.303844 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.303814 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:08.668482 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:14:08.668454 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37df1d28_38bf_417a_bfd3_83ceef6ed7ba.slice/crio-260571881ecf0d5867f90bdc93e7004a73c7d880d4e672fd17d91afe93e9843d WatchSource:0}: Error finding container 260571881ecf0d5867f90bdc93e7004a73c7d880d4e672fd17d91afe93e9843d: Status 404 returned error can't find the container with id 260571881ecf0d5867f90bdc93e7004a73c7d880d4e672fd17d91afe93e9843d Oct 31 20:14:08.670673 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.670641 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd"] Oct 31 20:14:08.931893 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.931814 6481 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:14:08.931893 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:08.931867 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:14:09.167686 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:09.167648 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" event={"ID":"37df1d28-38bf-417a-bfd3-83ceef6ed7ba","Type":"ContainerStarted","Data":"0612611ccfffcc5cfb23103619e8907be93eaf2a567fc8faef714a4ae849aa23"} Oct 31 20:14:09.168104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:09.167714 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" event={"ID":"37df1d28-38bf-417a-bfd3-83ceef6ed7ba","Type":"ContainerStarted","Data":"260571881ecf0d5867f90bdc93e7004a73c7d880d4e672fd17d91afe93e9843d"} Oct 31 20:14:09.284373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:09.284317 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" podStartSLOduration=2.284300858 podStartE2EDuration="2.284300858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:14:09.282798634 +0000 UTC m=+3420.523021764" watchObservedRunningTime="2026-10-31 20:14:09.284300858 +0000 UTC m=+3420.524523994" Oct 31 20:14:10.174063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:10.174029 6481 generic.go:356] "Generic (PLEG): container finished" podID="37df1d28-38bf-417a-bfd3-83ceef6ed7ba" containerID="0612611ccfffcc5cfb23103619e8907be93eaf2a567fc8faef714a4ae849aa23" exitCode=0 Oct 31 20:14:10.174453 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:10.174073 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" event={"ID":"37df1d28-38bf-417a-bfd3-83ceef6ed7ba","Type":"ContainerDied","Data":"0612611ccfffcc5cfb23103619e8907be93eaf2a567fc8faef714a4ae849aa23"} Oct 31 20:14:11.313542 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.313521 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:11.433908 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.433828 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-openshift-ca\") pod \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " Oct 31 20:14:11.434099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.433905 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-xks-ca\") pod \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " Oct 31 20:14:11.434099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.433969 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-kube-api-access-dnzr8\" (UniqueName: \"kubernetes.io/projected/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-kube-api-access-dnzr8\") pod \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\" (UID: \"37df1d28-38bf-417a-bfd3-83ceef6ed7ba\") " Oct 31 20:14:11.434302 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.434226 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-openshift-ca" pod "37df1d28-38bf-417a-bfd3-83ceef6ed7ba" (UID: "37df1d28-38bf-417a-bfd3-83ceef6ed7ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:11.434406 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.434385 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:11.436096 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.436072 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-kube-api-access-dnzr8" pod "37df1d28-38bf-417a-bfd3-83ceef6ed7ba" (UID: "37df1d28-38bf-417a-bfd3-83ceef6ed7ba"). InnerVolumeSpecName "kube-api-access-dnzr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:11.436498 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.436479 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-xks-ca" pod "37df1d28-38bf-417a-bfd3-83ceef6ed7ba" (UID: "37df1d28-38bf-417a-bfd3-83ceef6ed7ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:11.534941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.534908 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:11.534941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:11.534936 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnzr8\" (UniqueName: \"kubernetes.io/projected/37df1d28-38bf-417a-bfd3-83ceef6ed7ba-kube-api-access-dnzr8\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.190167 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.190136 6481 generic.go:356] "Generic (PLEG): container finished" podID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerID="21a0f438b96955476b0867592b092fd13bab50cc82c55c1a7bdea926e68e685a" exitCode=0 Oct 31 20:14:12.190303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.190211 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" event={"ID":"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5","Type":"ContainerDied","Data":"21a0f438b96955476b0867592b092fd13bab50cc82c55c1a7bdea926e68e685a"} Oct 31 20:14:12.190303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.190293 6481 scope.go:122] "RemoveContainer" containerID="243fce928ba1b7c0ddaada721e4139d003fc6d568ff8ac19fe10626a8a0779b7" Oct 31 20:14:12.191793 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.191738 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" Oct 31 20:14:12.191793 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.191752 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd" event={"ID":"37df1d28-38bf-417a-bfd3-83ceef6ed7ba","Type":"ContainerDied","Data":"260571881ecf0d5867f90bdc93e7004a73c7d880d4e672fd17d91afe93e9843d"} Oct 31 20:14:12.191908 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.191794 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="260571881ecf0d5867f90bdc93e7004a73c7d880d4e672fd17d91afe93e9843d" Oct 31 20:14:12.247980 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.247959 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 20:14:12.343857 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.343827 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-podinfo\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.343863 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-envoy\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.343892 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-credential-socket\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.343960 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-kube-api-access-gvfwq\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-kube-api-access-gvfwq\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344017 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-certs\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344094 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-socket\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344121 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istiod-ca-cert\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344170 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-token\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-token\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344204 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-data\") pod \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\" (UID: \"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5\") " Oct 31 20:14:12.344712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344296 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-credential-socket" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:12.344712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344376 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-certs" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:12.344712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344561 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-data" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:12.344712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344578 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-socket" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:12.344712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344630 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.344712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344650 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.344913 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.344741 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istiod-ca-cert" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:12.346103 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.346081 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-podinfo" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:14:12.346203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.346188 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-kube-api-access-gvfwq" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "kube-api-access-gvfwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:12.346435 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.346421 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-envoy" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:12.346610 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.346588 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-token" pod "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" (UID: "4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:12.445956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.445920 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.445956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.445947 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.445956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.445957 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.445956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.445966 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.446310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.445975 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvfwq\" (UniqueName: \"kubernetes.io/projected/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-kube-api-access-gvfwq\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.446310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.445985 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:12.446310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:12.446005 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:13.197638 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:13.197565 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" Oct 31 20:14:13.197638 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:13.197601 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7" event={"ID":"4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5","Type":"ContainerDied","Data":"7501991766a284100cce78b4d472a74aefbda237f4f232a85e0378b6438fca50"} Oct 31 20:14:13.197834 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:13.197653 6481 scope.go:122] "RemoveContainer" containerID="21a0f438b96955476b0867592b092fd13bab50cc82c55c1a7bdea926e68e685a" Oct 31 20:14:13.420276 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:13.420246 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7"] Oct 31 20:14:13.471290 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:13.471259 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6eeb83-openshift-default-7c5b66c55f-c9zp7"] Oct 31 20:14:14.877638 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:14.877602 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" path="/var/lib/kubelet/pods/4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5/volumes" Oct 31 20:14:19.979569 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.979490 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j"] Oct 31 20:14:19.979952 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.979905 6481 cpu_manager.go:414] "containerMap: removing container" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" Oct 31 20:14:19.979952 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.979932 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" Oct 31 20:14:19.980048 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.979954 6481 cpu_manager.go:414] "containerMap: removing container" podUID="37df1d28-38bf-417a-bfd3-83ceef6ed7ba" containerName="revoke-keys" Oct 31 20:14:19.980048 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.979959 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37df1d28-38bf-417a-bfd3-83ceef6ed7ba" containerName="revoke-keys" Oct 31 20:14:19.980048 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.980041 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" Oct 31 20:14:19.980146 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.980054 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="37df1d28-38bf-417a-bfd3-83ceef6ed7ba" containerName="revoke-keys" Oct 31 20:14:19.984597 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:19.984580 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.015471 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.015430 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.015638 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.015489 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fst8z\" (UniqueName: \"kubernetes.io/projected/3b0800e2-e40a-49d6-acd2-db082b08c0ec-kube-api-access-fst8z\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.015638 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.015587 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.033891 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.033850 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j"] Oct 31 20:14:20.083432 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.083394 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:14:20.083432 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.083427 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:14:20.116117 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.116086 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.116276 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.116138 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fst8z\" (UniqueName: \"kubernetes.io/projected/3b0800e2-e40a-49d6-acd2-db082b08c0ec-kube-api-access-fst8z\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.116276 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.116171 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.116846 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.116825 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.118424 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.118401 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.174094 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.174063 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fst8z\" (UniqueName: \"kubernetes.io/projected/3b0800e2-e40a-49d6-acd2-db082b08c0ec-kube-api-access-fst8z\") pod \"maas-api-revoke-sub-b4aa22f8-sj85j\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.297184 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.297095 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:20.665245 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:14:20.665214 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0800e2_e40a_49d6_acd2_db082b08c0ec.slice/crio-ccb51360e11b4f18684b080a7b733548278023cb304df84976f7c29c5453752d WatchSource:0}: Error finding container ccb51360e11b4f18684b080a7b733548278023cb304df84976f7c29c5453752d: Status 404 returned error can't find the container with id ccb51360e11b4f18684b080a7b733548278023cb304df84976f7c29c5453752d Oct 31 20:14:20.667842 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:20.667779 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j"] Oct 31 20:14:21.247028 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:21.246921 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" event={"ID":"3b0800e2-e40a-49d6-acd2-db082b08c0ec","Type":"ContainerStarted","Data":"37edaef36e3a8115076edaa832dd96608cc885895f97df4ed551d7ce3cfa042d"} Oct 31 20:14:21.247028 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:21.246979 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" event={"ID":"3b0800e2-e40a-49d6-acd2-db082b08c0ec","Type":"ContainerStarted","Data":"ccb51360e11b4f18684b080a7b733548278023cb304df84976f7c29c5453752d"} Oct 31 20:14:21.363796 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:21.363746 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" podStartSLOduration=2.363733366 podStartE2EDuration="2.363733366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:14:21.363130265 +0000 UTC m=+3432.603353396" watchObservedRunningTime="2026-10-31 20:14:21.363733366 +0000 UTC m=+3432.603956494" Oct 31 20:14:22.252625 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:22.252588 6481 generic.go:356] "Generic (PLEG): container finished" podID="3b0800e2-e40a-49d6-acd2-db082b08c0ec" containerID="37edaef36e3a8115076edaa832dd96608cc885895f97df4ed551d7ce3cfa042d" exitCode=0 Oct 31 20:14:22.253041 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:22.252683 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" event={"ID":"3b0800e2-e40a-49d6-acd2-db082b08c0ec","Type":"ContainerDied","Data":"37edaef36e3a8115076edaa832dd96608cc885895f97df4ed551d7ce3cfa042d"} Oct 31 20:14:23.387819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.387792 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:23.459083 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.459049 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0800e2-e40a-49d6-acd2-db082b08c0ec-kube-api-access-fst8z\" (UniqueName: \"kubernetes.io/projected/3b0800e2-e40a-49d6-acd2-db082b08c0ec-kube-api-access-fst8z\") pod \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " Oct 31 20:14:23.459237 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.459137 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-openshift-ca\") pod \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " Oct 31 20:14:23.459237 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.459208 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-xks-ca\") pod \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\" (UID: \"3b0800e2-e40a-49d6-acd2-db082b08c0ec\") " Oct 31 20:14:23.459490 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.459465 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-openshift-ca" pod "3b0800e2-e40a-49d6-acd2-db082b08c0ec" (UID: "3b0800e2-e40a-49d6-acd2-db082b08c0ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:23.461145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.461118 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0800e2-e40a-49d6-acd2-db082b08c0ec-kube-api-access-fst8z" pod "3b0800e2-e40a-49d6-acd2-db082b08c0ec" (UID: "3b0800e2-e40a-49d6-acd2-db082b08c0ec"). InnerVolumeSpecName "kube-api-access-fst8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:23.461248 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.461150 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-xks-ca" pod "3b0800e2-e40a-49d6-acd2-db082b08c0ec" (UID: "3b0800e2-e40a-49d6-acd2-db082b08c0ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:23.560946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.560878 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:23.560946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.560906 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0800e2-e40a-49d6-acd2-db082b08c0ec-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:23.560946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:23.560916 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fst8z\" (UniqueName: \"kubernetes.io/projected/3b0800e2-e40a-49d6-acd2-db082b08c0ec-kube-api-access-fst8z\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:24.265148 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:24.265106 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" event={"ID":"3b0800e2-e40a-49d6-acd2-db082b08c0ec","Type":"ContainerDied","Data":"ccb51360e11b4f18684b080a7b733548278023cb304df84976f7c29c5453752d"} Oct 31 20:14:24.265357 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:24.265169 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb51360e11b4f18684b080a7b733548278023cb304df84976f7c29c5453752d" Oct 31 20:14:24.265357 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:24.265121 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j" Oct 31 20:14:27.323779 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:14:27.323746 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:14:36.957541 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:36.957508 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd"] Oct 31 20:14:37.008597 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:37.008569 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-093b97-qnprd"] Oct 31 20:14:38.877931 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:38.877898 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37df1d28-38bf-417a-bfd3-83ceef6ed7ba" path="/var/lib/kubelet/pods/37df1d28-38bf-417a-bfd3-83ceef6ed7ba/volumes" Oct 31 20:14:43.513796 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:43.513754 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k"] Oct 31 20:14:43.514304 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:43.514032 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" containerID="cri-o://ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a" gracePeriod=30 Oct 31 20:14:48.723704 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.723670 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69947f5d84-nnkd9"] Oct 31 20:14:48.724225 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.723893 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-69947f5d84-nnkd9" podUID="3e6e5bb8-1043-4861-8037-a3b781de0cf7" containerName="manager" containerID="cri-o://a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35" gracePeriod=30 Oct 31 20:14:48.764537 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.764411 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 20:14:48.906583 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906541 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-certs\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.906797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906611 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99026ac1-6520-413a-aa94-86e7f85f5578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026ac1-6520-413a-aa94-86e7f85f5578-istio-podinfo\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.906797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906655 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-socket\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.906797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906681 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-credential-socket\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.906797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906742 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-envoy\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.907035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906818 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99026ac1-6520-413a-aa94-86e7f85f5578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026ac1-6520-413a-aa94-86e7f85f5578-istiod-ca-cert\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.907035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906849 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-istio-token\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-istio-token\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.907035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906899 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-data\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.907035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906936 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-kube-api-access-fdrm5\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-kube-api-access-fdrm5\") pod \"99026ac1-6520-413a-aa94-86e7f85f5578\" (UID: \"99026ac1-6520-413a-aa94-86e7f85f5578\") " Oct 31 20:14:48.907392 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.906843 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-certs" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:48.907392 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.907050 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-credential-socket" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:48.907524 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.907444 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-socket" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:48.907841 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.907791 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-data" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:48.908389 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.907981 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99026ac1-6520-413a-aa94-86e7f85f5578-istiod-ca-cert" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:14:48.909754 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.909727 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99026ac1-6520-413a-aa94-86e7f85f5578-istio-podinfo" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:14:48.910220 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.910193 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-istio-token" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:48.910321 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.910286 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-kube-api-access-fdrm5" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "kube-api-access-fdrm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:48.910692 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.910667 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-envoy" pod "99026ac1-6520-413a-aa94-86e7f85f5578" (UID: "99026ac1-6520-413a-aa94-86e7f85f5578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:14:48.957211 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:48.956701 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 20:14:49.008781 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008742 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026ac1-6520-413a-aa94-86e7f85f5578-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.008781 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008776 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.008781 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008788 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.009101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008800 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.009101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008812 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026ac1-6520-413a-aa94-86e7f85f5578-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.009101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008823 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.009101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008837 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.009101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008850 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrm5\" (UniqueName: \"kubernetes.io/projected/99026ac1-6520-413a-aa94-86e7f85f5578-kube-api-access-fdrm5\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.009101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.008866 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026ac1-6520-413a-aa94-86e7f85f5578-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.109563 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.109484 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6e5bb8-1043-4861-8037-a3b781de0cf7-kube-api-access-kpmxb\" (UniqueName: \"kubernetes.io/projected/3e6e5bb8-1043-4861-8037-a3b781de0cf7-kube-api-access-kpmxb\") pod \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " Oct 31 20:14:49.109563 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.109561 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-webhook-cert\") pod \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " Oct 31 20:14:49.109794 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.109608 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-metrics-certs\") pod \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\" (UID: \"3e6e5bb8-1043-4861-8037-a3b781de0cf7\") " Oct 31 20:14:49.111881 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.111846 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-metrics-certs" pod "3e6e5bb8-1043-4861-8037-a3b781de0cf7" (UID: "3e6e5bb8-1043-4861-8037-a3b781de0cf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:49.112157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.112126 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-webhook-cert" pod "3e6e5bb8-1043-4861-8037-a3b781de0cf7" (UID: "3e6e5bb8-1043-4861-8037-a3b781de0cf7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:14:49.112246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.112164 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6e5bb8-1043-4861-8037-a3b781de0cf7-kube-api-access-kpmxb" pod "3e6e5bb8-1043-4861-8037-a3b781de0cf7" (UID: "3e6e5bb8-1043-4861-8037-a3b781de0cf7"). InnerVolumeSpecName "kube-api-access-kpmxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:14:49.210801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.210764 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpmxb\" (UniqueName: \"kubernetes.io/projected/3e6e5bb8-1043-4861-8037-a3b781de0cf7-kube-api-access-kpmxb\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.210801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.210794 6481 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-webhook-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.210801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.210805 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e5bb8-1043-4861-8037-a3b781de0cf7-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:14:49.384854 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.384769 6481 generic.go:356] "Generic (PLEG): container finished" podID="99026ac1-6520-413a-aa94-86e7f85f5578" containerID="ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a" exitCode=0 Oct 31 20:14:49.384854 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.384817 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" Oct 31 20:14:49.385137 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.384849 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" event={"ID":"99026ac1-6520-413a-aa94-86e7f85f5578","Type":"ContainerDied","Data":"ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a"} Oct 31 20:14:49.385137 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.384901 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k" event={"ID":"99026ac1-6520-413a-aa94-86e7f85f5578","Type":"ContainerDied","Data":"c81b3a892ba0ea31ac3319f38fb7821e70a4eb9446da881338d323326d6eec42"} Oct 31 20:14:49.385137 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.384932 6481 scope.go:122] "RemoveContainer" containerID="ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a" Oct 31 20:14:49.386265 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.386242 6481 generic.go:356] "Generic (PLEG): container finished" podID="3e6e5bb8-1043-4861-8037-a3b781de0cf7" containerID="a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35" exitCode=0 Oct 31 20:14:49.386374 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.386283 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-nnkd9" Oct 31 20:14:49.386374 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.386302 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-nnkd9" event={"ID":"3e6e5bb8-1043-4861-8037-a3b781de0cf7","Type":"ContainerDied","Data":"a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35"} Oct 31 20:14:49.386374 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.386340 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-nnkd9" event={"ID":"3e6e5bb8-1043-4861-8037-a3b781de0cf7","Type":"ContainerDied","Data":"2bf130535b4a7ed9645d6eb51600133607781cfe1ce9d1f2ea7ad82f0c59cd1b"} Oct 31 20:14:49.399806 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.396762 6481 scope.go:122] "RemoveContainer" containerID="ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42" Oct 31 20:14:49.408484 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.408464 6481 scope.go:122] "RemoveContainer" containerID="ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a" Oct 31 20:14:49.408727 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:14:49.408707 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a\": container with ID starting with ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a not found: ID does not exist" containerID="ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a" Oct 31 20:14:49.408781 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.408736 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a"} err="failed to get container status \"ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a\": rpc error: code = NotFound desc = could not find container \"ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a\": container with ID starting with ff3223e0a44a74ab698075da24be64ff144d923884ce2313f49263c6c2f7858a not found: ID does not exist" Oct 31 20:14:49.408781 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.408759 6481 scope.go:122] "RemoveContainer" containerID="ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42" Oct 31 20:14:49.409153 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:14:49.409125 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42\": container with ID starting with ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42 not found: ID does not exist" containerID="ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42" Oct 31 20:14:49.409197 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.409172 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42"} err="failed to get container status \"ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42\": rpc error: code = NotFound desc = could not find container \"ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42\": container with ID starting with ae5b718219bc83319a5360e1124b5803ee9e9249522b4514550c90b659c51b42 not found: ID does not exist" Oct 31 20:14:49.409234 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.409207 6481 scope.go:122] "RemoveContainer" containerID="a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35" Oct 31 20:14:49.417646 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.417626 6481 scope.go:122] "RemoveContainer" containerID="a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35" Oct 31 20:14:49.417886 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:14:49.417870 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35\": container with ID starting with a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35 not found: ID does not exist" containerID="a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35" Oct 31 20:14:49.417935 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.417891 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35"} err="failed to get container status \"a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35\": rpc error: code = NotFound desc = could not find container \"a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35\": container with ID starting with a097ac387c65e524cb24035a789e969c6378672f0087ff02447415d2111f6e35 not found: ID does not exist" Oct 31 20:14:49.554074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.554045 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k"] Oct 31 20:14:49.558605 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.558583 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-093b97-openshift-default-7898b77ddf-6zv5k"] Oct 31 20:14:49.672712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.672625 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69947f5d84-nnkd9"] Oct 31 20:14:49.676245 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:49.676213 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69947f5d84-nnkd9"] Oct 31 20:14:50.548014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.547967 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69947f5d84-rqp7s"] Oct 31 20:14:50.548413 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548370 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3b0800e2-e40a-49d6-acd2-db082b08c0ec" containerName="revoke-keys" Oct 31 20:14:50.548413 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548381 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0800e2-e40a-49d6-acd2-db082b08c0ec" containerName="revoke-keys" Oct 31 20:14:50.548413 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548403 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3e6e5bb8-1043-4861-8037-a3b781de0cf7" containerName="manager" Oct 31 20:14:50.548413 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548408 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6e5bb8-1043-4861-8037-a3b781de0cf7" containerName="manager" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548420 6481 cpu_manager.go:414] "containerMap: removing container" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548426 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548431 6481 cpu_manager.go:414] "containerMap: removing container" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548436 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548489 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0800e2-e40a-49d6-acd2-db082b08c0ec" containerName="revoke-keys" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548500 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6e5bb8-1043-4861-8037-a3b781de0cf7" containerName="manager" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548509 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" Oct 31 20:14:50.548540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.548514 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f41f49d-f53c-4dc9-a9d7-8b242c3c49f5" containerName="istio-proxy" Oct 31 20:14:50.552864 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.552847 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.657906 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.657865 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 20:14:50.657906 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.657882 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rz6ng\"" Oct 31 20:14:50.658135 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.657912 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 20:14:50.658135 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.658063 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 20:14:50.659590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.659570 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69947f5d84-rqp7s"] Oct 31 20:14:50.725178 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.725146 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-webhook-cert\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.725178 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.725178 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-metrics-certs\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.725404 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.725203 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25qq\" (UniqueName: \"kubernetes.io/projected/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-kube-api-access-b25qq\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.826559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.826477 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-webhook-cert\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.826559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.826510 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-metrics-certs\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.826559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.826553 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25qq\" (UniqueName: \"kubernetes.io/projected/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-kube-api-access-b25qq\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.828958 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.828932 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-webhook-cert\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.829080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.828990 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-metrics-certs\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:50.877953 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.877919 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6e5bb8-1043-4861-8037-a3b781de0cf7" path="/var/lib/kubelet/pods/3e6e5bb8-1043-4861-8037-a3b781de0cf7/volumes" Oct 31 20:14:50.878422 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.878407 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" path="/var/lib/kubelet/pods/99026ac1-6520-413a-aa94-86e7f85f5578/volumes" Oct 31 20:14:50.885914 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:50.885882 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25qq\" (UniqueName: \"kubernetes.io/projected/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-kube-api-access-b25qq\") pod \"maas-controller-69947f5d84-rqp7s\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:51.164782 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:51.164707 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:51.538589 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:14:51.538563 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf8de7c_3d29_429b_bab8_b5a8b660c71a.slice/crio-ba5e1a2ec43345b127af36a15ad6299b553f0e58bf95602b060258a3e115a990 WatchSource:0}: Error finding container ba5e1a2ec43345b127af36a15ad6299b553f0e58bf95602b060258a3e115a990: Status 404 returned error can't find the container with id ba5e1a2ec43345b127af36a15ad6299b553f0e58bf95602b060258a3e115a990 Oct 31 20:14:51.539832 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:51.539814 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:14:51.540803 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:51.540780 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69947f5d84-rqp7s"] Oct 31 20:14:52.406208 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:52.406168 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-rqp7s" event={"ID":"ecf8de7c-3d29-429b-bab8-b5a8b660c71a","Type":"ContainerStarted","Data":"ba5e1a2ec43345b127af36a15ad6299b553f0e58bf95602b060258a3e115a990"} Oct 31 20:14:53.416963 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:53.416902 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-rqp7s" event={"ID":"ecf8de7c-3d29-429b-bab8-b5a8b660c71a","Type":"ContainerStarted","Data":"0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e"} Oct 31 20:14:53.417530 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:53.417288 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:14:53.541832 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:14:53.541779 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-69947f5d84-rqp7s" podStartSLOduration=2.644516248 podStartE2EDuration="3.541765485s" totalImagesPullingTime="897.249237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:14:53.540882654 +0000 UTC m=+3464.781105782" watchObservedRunningTime="2026-10-31 20:14:53.541765485 +0000 UTC m=+3464.781988591" Oct 31 20:14:57.325228 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:14:57.325184 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:15:00.165598 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.165560 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n"] Oct 31 20:15:00.166047 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.166034 6481 cpu_manager.go:414] "containerMap: removing container" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" Oct 31 20:15:00.166100 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.166049 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" Oct 31 20:15:00.166135 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.166110 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="99026ac1-6520-413a-aa94-86e7f85f5578" containerName="istio-proxy" Oct 31 20:15:00.169335 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.169317 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.204833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.204798 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f252d\" (UniqueName: \"kubernetes.io/projected/35fa912c-dfc3-4c65-8254-285eee2e83bd-kube-api-access-f252d\") pod \"maas-api-key-cleanup-29891295-8fq7n\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.204977 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.204846 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fa912c-dfc3-4c65-8254-285eee2e83bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-8fq7n\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.221768 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.221737 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n"] Oct 31 20:15:00.271673 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.271641 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:15:00.306198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.306159 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f252d\" (UniqueName: \"kubernetes.io/projected/35fa912c-dfc3-4c65-8254-285eee2e83bd-kube-api-access-f252d\") pod \"maas-api-key-cleanup-29891295-8fq7n\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.306403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.306206 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fa912c-dfc3-4c65-8254-285eee2e83bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-8fq7n\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.306863 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.306842 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fa912c-dfc3-4c65-8254-285eee2e83bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-8fq7n\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.364647 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.364611 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f252d\" (UniqueName: \"kubernetes.io/projected/35fa912c-dfc3-4c65-8254-285eee2e83bd-kube-api-access-f252d\") pod \"maas-api-key-cleanup-29891295-8fq7n\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.482500 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.482467 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:00.863589 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:15:00.863555 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fa912c_dfc3_4c65_8254_285eee2e83bd.slice/crio-d953f6f5ff15748d07c2b437d4c99725caf643301268a3f728a427edcd153276 WatchSource:0}: Error finding container d953f6f5ff15748d07c2b437d4c99725caf643301268a3f728a427edcd153276: Status 404 returned error can't find the container with id d953f6f5ff15748d07c2b437d4c99725caf643301268a3f728a427edcd153276 Oct 31 20:15:00.865191 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:00.865167 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n"] Oct 31 20:15:01.458968 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:01.458875 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" event={"ID":"35fa912c-dfc3-4c65-8254-285eee2e83bd","Type":"ContainerStarted","Data":"1894f71508d9b6fc7e85ff14cf64ac64e8e91ac0cbe64a66af6c9b27518adce5"} Oct 31 20:15:01.458968 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:01.458959 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" event={"ID":"35fa912c-dfc3-4c65-8254-285eee2e83bd","Type":"ContainerStarted","Data":"d953f6f5ff15748d07c2b437d4c99725caf643301268a3f728a427edcd153276"} Oct 31 20:15:01.575352 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:01.575294 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" podStartSLOduration=1.575277568 podStartE2EDuration="1.575277568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:01.573699305 +0000 UTC m=+3472.813922434" watchObservedRunningTime="2026-10-31 20:15:01.575277568 +0000 UTC m=+3472.815500696" Oct 31 20:15:02.465414 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:02.465382 6481 generic.go:356] "Generic (PLEG): container finished" podID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerID="1894f71508d9b6fc7e85ff14cf64ac64e8e91ac0cbe64a66af6c9b27518adce5" exitCode=60 Oct 31 20:15:02.465785 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:02.465419 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" event={"ID":"35fa912c-dfc3-4c65-8254-285eee2e83bd","Type":"ContainerDied","Data":"1894f71508d9b6fc7e85ff14cf64ac64e8e91ac0cbe64a66af6c9b27518adce5"} Oct 31 20:15:02.465785 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:02.465676 6481 scope.go:122] "RemoveContainer" containerID="1894f71508d9b6fc7e85ff14cf64ac64e8e91ac0cbe64a66af6c9b27518adce5" Oct 31 20:15:03.472323 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:03.472287 6481 generic.go:356] "Generic (PLEG): container finished" podID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerID="f2592fe8ff794bcc37c5453c1f8d36bf3b9bdf84ee4efd88d1a404305bb54a4b" exitCode=60 Oct 31 20:15:03.472769 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:03.472370 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" event={"ID":"35fa912c-dfc3-4c65-8254-285eee2e83bd","Type":"ContainerDied","Data":"f2592fe8ff794bcc37c5453c1f8d36bf3b9bdf84ee4efd88d1a404305bb54a4b"} Oct 31 20:15:03.472769 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:03.472425 6481 scope.go:122] "RemoveContainer" containerID="1894f71508d9b6fc7e85ff14cf64ac64e8e91ac0cbe64a66af6c9b27518adce5" Oct 31 20:15:03.472769 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:03.472667 6481 scope.go:122] "RemoveContainer" containerID="f2592fe8ff794bcc37c5453c1f8d36bf3b9bdf84ee4efd88d1a404305bb54a4b" Oct 31 20:15:03.473015 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:15:03.472947 6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-8fq7n_odh-ai-gateway-infra(35fa912c-dfc3-4c65-8254-285eee2e83bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" Oct 31 20:15:04.424090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:04.424053 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:15:04.478574 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:04.478550 6481 scope.go:122] "RemoveContainer" containerID="f2592fe8ff794bcc37c5453c1f8d36bf3b9bdf84ee4efd88d1a404305bb54a4b" Oct 31 20:15:04.478956 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:15:04.478788 6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-8fq7n_odh-ai-gateway-infra(35fa912c-dfc3-4c65-8254-285eee2e83bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" Oct 31 20:15:16.872192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:16.872161 6481 scope.go:122] "RemoveContainer" containerID="f2592fe8ff794bcc37c5453c1f8d36bf3b9bdf84ee4efd88d1a404305bb54a4b" Oct 31 20:15:17.538688 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:17.538648 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" event={"ID":"35fa912c-dfc3-4c65-8254-285eee2e83bd","Type":"ContainerStarted","Data":"6f2ee1dedfeed73f4d354255b1cc4822050ef05f8262349f488b891750d33b4d"} Oct 31 20:15:18.545236 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:18.545196 6481 generic.go:356] "Generic (PLEG): container finished" podID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerID="6f2ee1dedfeed73f4d354255b1cc4822050ef05f8262349f488b891750d33b4d" exitCode=60 Oct 31 20:15:18.545648 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:18.545268 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" event={"ID":"35fa912c-dfc3-4c65-8254-285eee2e83bd","Type":"ContainerDied","Data":"6f2ee1dedfeed73f4d354255b1cc4822050ef05f8262349f488b891750d33b4d"} Oct 31 20:15:18.545648 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:18.545328 6481 scope.go:122] "RemoveContainer" containerID="f2592fe8ff794bcc37c5453c1f8d36bf3b9bdf84ee4efd88d1a404305bb54a4b" Oct 31 20:15:18.545648 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:18.545621 6481 scope.go:122] "RemoveContainer" containerID="6f2ee1dedfeed73f4d354255b1cc4822050ef05f8262349f488b891750d33b4d" Oct 31 20:15:18.545855 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:15:18.545833 6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-8fq7n_odh-ai-gateway-infra(35fa912c-dfc3-4c65-8254-285eee2e83bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" Oct 31 20:15:18.661674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:18.661634 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n"] Oct 31 20:15:19.679960 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:19.679935 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:19.779905 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:19.779875 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fa912c-dfc3-4c65-8254-285eee2e83bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fa912c-dfc3-4c65-8254-285eee2e83bd-maas-api-service-ca\") pod \"35fa912c-dfc3-4c65-8254-285eee2e83bd\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " Oct 31 20:15:19.780112 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:19.780038 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fa912c-dfc3-4c65-8254-285eee2e83bd-kube-api-access-f252d\" (UniqueName: \"kubernetes.io/projected/35fa912c-dfc3-4c65-8254-285eee2e83bd-kube-api-access-f252d\") pod \"35fa912c-dfc3-4c65-8254-285eee2e83bd\" (UID: \"35fa912c-dfc3-4c65-8254-285eee2e83bd\") " Oct 31 20:15:19.780260 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:19.780237 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fa912c-dfc3-4c65-8254-285eee2e83bd-maas-api-service-ca" pod "35fa912c-dfc3-4c65-8254-285eee2e83bd" (UID: "35fa912c-dfc3-4c65-8254-285eee2e83bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:15:19.780328 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:19.780312 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fa912c-dfc3-4c65-8254-285eee2e83bd-maas-api-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:15:19.782062 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:19.782037 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fa912c-dfc3-4c65-8254-285eee2e83bd-kube-api-access-f252d" pod "35fa912c-dfc3-4c65-8254-285eee2e83bd" (UID: "35fa912c-dfc3-4c65-8254-285eee2e83bd"). InnerVolumeSpecName "kube-api-access-f252d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:19.880985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:19.880909 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f252d\" (UniqueName: \"kubernetes.io/projected/35fa912c-dfc3-4c65-8254-285eee2e83bd-kube-api-access-f252d\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:15:20.556193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:20.556160 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" Oct 31 20:15:20.556373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:20.556206 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n" event={"ID":"35fa912c-dfc3-4c65-8254-285eee2e83bd","Type":"ContainerDied","Data":"d953f6f5ff15748d07c2b437d4c99725caf643301268a3f728a427edcd153276"} Oct 31 20:15:20.556373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:20.556268 6481 scope.go:122] "RemoveContainer" containerID="6f2ee1dedfeed73f4d354255b1cc4822050ef05f8262349f488b891750d33b4d" Oct 31 20:15:20.730507 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:20.730479 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n"] Oct 31 20:15:20.781557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:20.781525 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-8fq7n"] Oct 31 20:15:20.878445 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:20.878367 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" path="/var/lib/kubelet/pods/35fa912c-dfc3-4c65-8254-285eee2e83bd/volumes" Oct 31 20:15:24.676391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:24.676352 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69947f5d84-rqp7s"] Oct 31 20:15:24.676833 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:24.676561 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-69947f5d84-rqp7s" podUID="ecf8de7c-3d29-429b-bab8-b5a8b660c71a" containerName="manager" containerID="cri-o://0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e" gracePeriod=30 Oct 31 20:15:24.915504 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:24.915478 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:15:25.031252 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.031125 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-webhook-cert\") pod \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " Oct 31 20:15:25.031438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.031333 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-kube-api-access-b25qq\" (UniqueName: \"kubernetes.io/projected/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-kube-api-access-b25qq\") pod \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " Oct 31 20:15:25.031438 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.031403 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-metrics-certs\") pod \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\" (UID: \"ecf8de7c-3d29-429b-bab8-b5a8b660c71a\") " Oct 31 20:15:25.034338 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.034298 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-webhook-cert" pod "ecf8de7c-3d29-429b-bab8-b5a8b660c71a" (UID: "ecf8de7c-3d29-429b-bab8-b5a8b660c71a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:25.034518 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.034488 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-metrics-certs" pod "ecf8de7c-3d29-429b-bab8-b5a8b660c71a" (UID: "ecf8de7c-3d29-429b-bab8-b5a8b660c71a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:15:25.034845 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.034787 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-kube-api-access-b25qq" pod "ecf8de7c-3d29-429b-bab8-b5a8b660c71a" (UID: "ecf8de7c-3d29-429b-bab8-b5a8b660c71a"). InnerVolumeSpecName "kube-api-access-b25qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:15:25.132412 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.132382 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:15:25.132412 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.132406 6481 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-webhook-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:15:25.132412 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.132416 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25qq\" (UniqueName: \"kubernetes.io/projected/ecf8de7c-3d29-429b-bab8-b5a8b660c71a-kube-api-access-b25qq\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:15:25.579235 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.579196 6481 generic.go:356] "Generic (PLEG): container finished" podID="ecf8de7c-3d29-429b-bab8-b5a8b660c71a" containerID="0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e" exitCode=0 Oct 31 20:15:25.579410 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.579260 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-rqp7s" Oct 31 20:15:25.579410 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.579281 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-rqp7s" event={"ID":"ecf8de7c-3d29-429b-bab8-b5a8b660c71a","Type":"ContainerDied","Data":"0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e"} Oct 31 20:15:25.579410 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.579362 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-rqp7s" event={"ID":"ecf8de7c-3d29-429b-bab8-b5a8b660c71a","Type":"ContainerDied","Data":"ba5e1a2ec43345b127af36a15ad6299b553f0e58bf95602b060258a3e115a990"} Oct 31 20:15:25.579410 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.579386 6481 scope.go:122] "RemoveContainer" containerID="0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e" Oct 31 20:15:25.588442 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.588426 6481 scope.go:122] "RemoveContainer" containerID="0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e" Oct 31 20:15:25.588680 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:15:25.588662 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e\": container with ID starting with 0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e not found: ID does not exist" containerID="0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e" Oct 31 20:15:25.588726 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.588688 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e"} err="failed to get container status \"0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e\": rpc error: code = NotFound desc = could not find container \"0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e\": container with ID starting with 0a6c1ac1e2f1f03a6062fa21bd662af52d8d9f39b3a3271b561d068c6144447e not found: ID does not exist" Oct 31 20:15:25.748225 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.748192 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69947f5d84-rqp7s"] Oct 31 20:15:25.751224 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751195 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls"] Oct 31 20:15:25.751561 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751549 6481 cpu_manager.go:414] "containerMap: removing container" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:25.751613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751562 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:25.751613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751575 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ecf8de7c-3d29-429b-bab8-b5a8b660c71a" containerName="manager" Oct 31 20:15:25.751613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751580 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf8de7c-3d29-429b-bab8-b5a8b660c71a" containerName="manager" Oct 31 20:15:25.751613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751590 6481 cpu_manager.go:414] "containerMap: removing container" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:25.751613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751595 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:25.751790 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751660 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:25.751790 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751674 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf8de7c-3d29-429b-bab8-b5a8b660c71a" containerName="manager" Oct 31 20:15:25.751790 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.751680 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:25.756028 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.756013 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.799213 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.799189 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69947f5d84-rqp7s"] Oct 31 20:15:25.810675 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.810642 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls"] Oct 31 20:15:25.838443 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838382 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838443 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838418 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838443 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838439 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838650 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838545 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838650 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838601 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838650 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838624 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxpw\" (UniqueName: \"kubernetes.io/projected/20b9145e-b5bb-4776-a30f-a904b9e01370-kube-api-access-tpxpw\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838650 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838647 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838837 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838720 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.838837 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.838770 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9145e-b5bb-4776-a30f-a904b9e01370-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.939814 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.939783 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.939983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.939826 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.939983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.939848 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxpw\" (UniqueName: \"kubernetes.io/projected/20b9145e-b5bb-4776-a30f-a904b9e01370-kube-api-access-tpxpw\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.939983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.939881 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.939983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.939922 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.939983 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.939978 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9145e-b5bb-4776-a30f-a904b9e01370-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940279 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940054 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940279 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940101 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940279 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940140 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940428 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940388 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940491 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940428 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940548 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940524 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940624 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940603 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.940850 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.940829 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9145e-b5bb-4776-a30f-a904b9e01370-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.942199 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.942177 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.942477 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.942459 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.998145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.998113 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9145e-b5bb-4776-a30f-a904b9e01370-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:25.998299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:25.998227 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxpw\" (UniqueName: \"kubernetes.io/projected/20b9145e-b5bb-4776-a30f-a904b9e01370-kube-api-access-tpxpw\") pod \"maas-default-gateway-openshift-default-58d898bd55-f9fls\" (UID: \"20b9145e-b5bb-4776-a30f-a904b9e01370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:26.067978 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.067939 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:26.483516 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:15:26.483485 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b9145e_b5bb_4776_a30f_a904b9e01370.slice/crio-1d9ba645e5b2cb7998018c52437769e7a33f1448d74452db74551cb0800688eb WatchSource:0}: Error finding container 1d9ba645e5b2cb7998018c52437769e7a33f1448d74452db74551cb0800688eb: Status 404 returned error can't find the container with id 1d9ba645e5b2cb7998018c52437769e7a33f1448d74452db74551cb0800688eb Oct 31 20:15:26.485072 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.484982 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls"] Oct 31 20:15:26.485718 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.485690 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:15:26.485784 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.485751 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:15:26.485821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.485783 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:15:26.588304 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.588267 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" event={"ID":"20b9145e-b5bb-4776-a30f-a904b9e01370","Type":"ContainerStarted","Data":"9a34c2b265001107f2e760ce35125dd1d7a3052dfac2f1e657bd89cf316039de"} Oct 31 20:15:26.588490 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.588317 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" event={"ID":"20b9145e-b5bb-4776-a30f-a904b9e01370","Type":"ContainerStarted","Data":"1d9ba645e5b2cb7998018c52437769e7a33f1448d74452db74551cb0800688eb"} Oct 31 20:15:26.706962 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.706905 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podStartSLOduration=1.706891298 podStartE2EDuration="1.706891298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:26.70594497 +0000 UTC m=+3497.946168099" watchObservedRunningTime="2026-10-31 20:15:26.706891298 +0000 UTC m=+3497.947114406" Oct 31 20:15:26.878821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:26.878743 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf8de7c-3d29-429b-bab8-b5a8b660c71a" path="/var/lib/kubelet/pods/ecf8de7c-3d29-429b-bab8-b5a8b660c71a/volumes" Oct 31 20:15:27.082019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.073281 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:27.082019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.074724 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 20:15:27.082019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.074793 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 20:15:27.090166 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.090130 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69947f5d84-8lqlr"] Oct 31 20:15:27.090761 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.090743 6481 cpu_manager.go:414] "containerMap: removing container" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:27.090818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.090764 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:27.090912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.090868 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fa912c-dfc3-4c65-8254-285eee2e83bd" containerName="cleanup" Oct 31 20:15:27.094505 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.094480 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.144348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.144280 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69947f5d84-8lqlr"] Oct 31 20:15:27.148064 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.148042 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 20:15:27.151421 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.151396 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72efb8b4-3301-404e-b4d1-a43275992fd8-metrics-certs\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.151507 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.151466 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72efb8b4-3301-404e-b4d1-a43275992fd8-webhook-cert\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.151563 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.151537 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwm7\" (UniqueName: \"kubernetes.io/projected/72efb8b4-3301-404e-b4d1-a43275992fd8-kube-api-access-8rwm7\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.193255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.193210 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 20:15:27.193255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.193233 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 20:15:27.193255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.193244 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rz6ng\"" Oct 31 20:15:27.252484 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.252445 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72efb8b4-3301-404e-b4d1-a43275992fd8-metrics-certs\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.252695 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.252530 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72efb8b4-3301-404e-b4d1-a43275992fd8-webhook-cert\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.252695 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.252552 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwm7\" (UniqueName: \"kubernetes.io/projected/72efb8b4-3301-404e-b4d1-a43275992fd8-kube-api-access-8rwm7\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.255092 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.255061 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72efb8b4-3301-404e-b4d1-a43275992fd8-webhook-cert\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.255205 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.255182 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72efb8b4-3301-404e-b4d1-a43275992fd8-metrics-certs\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.312913 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.312873 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwm7\" (UniqueName: \"kubernetes.io/projected/72efb8b4-3301-404e-b4d1-a43275992fd8-kube-api-access-8rwm7\") pod \"maas-controller-69947f5d84-8lqlr\" (UID: \"72efb8b4-3301-404e-b4d1-a43275992fd8\") " pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.325397 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:15:27.325360 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:15:27.408510 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.408416 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:27.792341 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:27.792291 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69947f5d84-8lqlr"] Oct 31 20:15:28.087505 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:28.087417 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 20:15:28.088117 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:28.088072 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 20:15:28.600544 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:28.600511 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-8lqlr" event={"ID":"72efb8b4-3301-404e-b4d1-a43275992fd8","Type":"ContainerStarted","Data":"6fe40901f5c51c6312300b2a3ff3202cd51bf4ba7d5033deb21bd0a9b09ecd39"} Oct 31 20:15:29.069025 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:29.068929 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 20:15:29.069632 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:29.069388 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 20:15:29.606886 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:29.606846 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69947f5d84-8lqlr" event={"ID":"72efb8b4-3301-404e-b4d1-a43275992fd8","Type":"ContainerStarted","Data":"e4ea563c60e3f2710b253aa6b89fe04dad34ff61345fd6becb2166d11bdae25b"} Oct 31 20:15:29.607334 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:29.607051 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:29.726129 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:29.726073 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-69947f5d84-8lqlr" podStartSLOduration=1.727612835 podStartE2EDuration="2.726058875s" totalImagesPullingTime="998.44604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:29.723887803 +0000 UTC m=+3500.964110943" watchObservedRunningTime="2026-10-31 20:15:29.726058875 +0000 UTC m=+3500.966282003" Oct 31 20:15:30.076426 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:30.076386 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 20:15:30.076727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:30.076693 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 20:15:31.084019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:31.072401 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 20:15:31.084019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:31.072462 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 20:15:32.083020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:32.077450 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 20:15:32.083020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:32.077533 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 20:15:33.068691 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:33.068651 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 20:15:33.069080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:33.068719 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 20:15:34.685541 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:34.685506 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:34.686039 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:34.685789 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:34.686818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:34.686798 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" Oct 31 20:15:40.613904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:15:40.613874 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69947f5d84-8lqlr" Oct 31 20:15:57.324718 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:15:57.324681 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:16:23.080390 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.080356 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr"] Oct 31 20:16:23.085091 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.085051 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.125938 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.125910 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.126086 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.125979 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgxl\" (UniqueName: \"kubernetes.io/projected/a934192d-c121-4aad-874d-71aa80a6a875-kube-api-access-lpgxl\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.126139 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.126080 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.136525 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.136496 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr"] Oct 31 20:16:23.185923 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.185867 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:16:23.186124 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.185958 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:16:23.227327 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.227294 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.227497 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.227372 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgxl\" (UniqueName: \"kubernetes.io/projected/a934192d-c121-4aad-874d-71aa80a6a875-kube-api-access-lpgxl\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.227497 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.227404 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.227977 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.227946 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.229673 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.229652 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.285816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.285787 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgxl\" (UniqueName: \"kubernetes.io/projected/a934192d-c121-4aad-874d-71aa80a6a875-kube-api-access-lpgxl\") pod \"maas-api-revoke-sub-c1e271a9-7gmmr\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.400876 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.400797 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:23.763652 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:16:23.763618 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda934192d_c121_4aad_874d_71aa80a6a875.slice/crio-b483d641651d5963e1e849c278afc89248261884ae9286d88837fad460510b98 WatchSource:0}: Error finding container b483d641651d5963e1e849c278afc89248261884ae9286d88837fad460510b98: Status 404 returned error can't find the container with id b483d641651d5963e1e849c278afc89248261884ae9286d88837fad460510b98 Oct 31 20:16:23.766894 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.766864 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr"] Oct 31 20:16:23.880535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:23.880501 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" event={"ID":"a934192d-c121-4aad-874d-71aa80a6a875","Type":"ContainerStarted","Data":"b483d641651d5963e1e849c278afc89248261884ae9286d88837fad460510b98"} Oct 31 20:16:24.886330 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:24.886295 6481 generic.go:356] "Generic (PLEG): container finished" podID="a934192d-c121-4aad-874d-71aa80a6a875" containerID="c464cfab0cdd186c582ac04ae4855e662f250899b673ee6f516a3908ed531020" exitCode=0 Oct 31 20:16:24.886737 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:24.886374 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" event={"ID":"a934192d-c121-4aad-874d-71aa80a6a875","Type":"ContainerDied","Data":"c464cfab0cdd186c582ac04ae4855e662f250899b673ee6f516a3908ed531020"} Oct 31 20:16:26.020376 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.020347 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:26.155907 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.155833 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-xks-ca\") pod \"a934192d-c121-4aad-874d-71aa80a6a875\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " Oct 31 20:16:26.156100 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.155940 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a934192d-c121-4aad-874d-71aa80a6a875-kube-api-access-lpgxl\" (UniqueName: \"kubernetes.io/projected/a934192d-c121-4aad-874d-71aa80a6a875-kube-api-access-lpgxl\") pod \"a934192d-c121-4aad-874d-71aa80a6a875\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " Oct 31 20:16:26.156100 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.156015 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-openshift-ca\") pod \"a934192d-c121-4aad-874d-71aa80a6a875\" (UID: \"a934192d-c121-4aad-874d-71aa80a6a875\") " Oct 31 20:16:26.156386 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.156362 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-openshift-ca" pod "a934192d-c121-4aad-874d-71aa80a6a875" (UID: "a934192d-c121-4aad-874d-71aa80a6a875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:16:26.158073 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.158020 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-xks-ca" pod "a934192d-c121-4aad-874d-71aa80a6a875" (UID: "a934192d-c121-4aad-874d-71aa80a6a875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:16:26.158175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.158081 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a934192d-c121-4aad-874d-71aa80a6a875-kube-api-access-lpgxl" pod "a934192d-c121-4aad-874d-71aa80a6a875" (UID: "a934192d-c121-4aad-874d-71aa80a6a875"). InnerVolumeSpecName "kube-api-access-lpgxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:16:26.257484 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.257449 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgxl\" (UniqueName: \"kubernetes.io/projected/a934192d-c121-4aad-874d-71aa80a6a875-kube-api-access-lpgxl\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:16:26.257484 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.257479 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:16:26.257484 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.257489 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a934192d-c121-4aad-874d-71aa80a6a875-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:16:26.896415 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.896386 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" Oct 31 20:16:26.896569 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.896410 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr" event={"ID":"a934192d-c121-4aad-874d-71aa80a6a875","Type":"ContainerDied","Data":"b483d641651d5963e1e849c278afc89248261884ae9286d88837fad460510b98"} Oct 31 20:16:26.896569 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:26.896458 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b483d641651d5963e1e849c278afc89248261884ae9286d88837fad460510b98" Oct 31 20:16:27.323979 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:16:27.323938 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:16:41.332057 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.332016 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx"] Oct 31 20:16:41.332985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.332956 6481 cpu_manager.go:414] "containerMap: removing container" podUID="a934192d-c121-4aad-874d-71aa80a6a875" containerName="revoke-keys" Oct 31 20:16:41.332985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.332986 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a934192d-c121-4aad-874d-71aa80a6a875" containerName="revoke-keys" Oct 31 20:16:41.333241 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.333226 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a934192d-c121-4aad-874d-71aa80a6a875" containerName="revoke-keys" Oct 31 20:16:41.337549 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.337519 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.508721 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.508683 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrxf\" (UniqueName: \"kubernetes.io/projected/14094e68-424c-4b92-b951-506da45efa7e-kube-api-access-tjrxf\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.508912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.508792 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14094e68-424c-4b92-b951-506da45efa7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.508912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.508852 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.508912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.508896 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.509140 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.508963 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.509140 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.509045 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14094e68-424c-4b92-b951-506da45efa7e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.509140 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.509084 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.509300 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.509143 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.509300 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.509235 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14094e68-424c-4b92-b951-506da45efa7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.532869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.532831 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx"] Oct 31 20:16:41.609823 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.609738 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14094e68-424c-4b92-b951-506da45efa7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.609823 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.609784 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.609837 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.609873 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.609923 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14094e68-424c-4b92-b951-506da45efa7e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.609959 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.609989 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.610051 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14094e68-424c-4b92-b951-506da45efa7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610414 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.610103 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrxf\" (UniqueName: \"kubernetes.io/projected/14094e68-424c-4b92-b951-506da45efa7e-kube-api-access-tjrxf\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610414 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.610275 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610414 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.610367 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610548 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.610524 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610548 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.610539 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.610962 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.610940 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14094e68-424c-4b92-b951-506da45efa7e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.612315 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.612288 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14094e68-424c-4b92-b951-506da45efa7e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.612456 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.612438 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14094e68-424c-4b92-b951-506da45efa7e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.668708 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.668670 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14094e68-424c-4b92-b951-506da45efa7e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.668902 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.668875 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrxf\" (UniqueName: \"kubernetes.io/projected/14094e68-424c-4b92-b951-506da45efa7e-kube-api-access-tjrxf\") pod \"maas-default-gateway-openshift-default-58d898bd55-6qfjx\" (UID: \"14094e68-424c-4b92-b951-506da45efa7e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:41.951221 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:41.951128 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:42.393881 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:42.393838 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx"] Oct 31 20:16:42.427246 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:16:42.427205 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14094e68_424c_4b92_b951_506da45efa7e.slice/crio-395ecfe4651583c9c6a5a17e7dbe2ea05e36da10e1068c01209bc59b89db7771 WatchSource:0}: Error finding container 395ecfe4651583c9c6a5a17e7dbe2ea05e36da10e1068c01209bc59b89db7771: Status 404 returned error can't find the container with id 395ecfe4651583c9c6a5a17e7dbe2ea05e36da10e1068c01209bc59b89db7771 Oct 31 20:16:42.430064 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:42.430022 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:16:42.430181 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:42.430108 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:16:42.430181 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:42.430148 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:16:42.987025 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:42.985515 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" event={"ID":"14094e68-424c-4b92-b951-506da45efa7e","Type":"ContainerStarted","Data":"ea446d85344abe982780757a2ff8bb22d52406c1f022d8168c4711656a7e550b"} Oct 31 20:16:42.987025 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:42.985586 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" event={"ID":"14094e68-424c-4b92-b951-506da45efa7e","Type":"ContainerStarted","Data":"395ecfe4651583c9c6a5a17e7dbe2ea05e36da10e1068c01209bc59b89db7771"} Oct 31 20:16:43.116019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:43.115309 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podStartSLOduration=2.115285753 podStartE2EDuration="2.115285753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:16:43.104448986 +0000 UTC m=+3574.344672129" watchObservedRunningTime="2026-10-31 20:16:43.115285753 +0000 UTC m=+3574.355508884" Oct 31 20:16:43.954017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:43.953412 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:43.954017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:43.953597 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 20:16:43.954017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:43.953657 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 20:16:44.952434 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:44.952401 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 20:16:44.952620 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:44.952454 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 20:16:45.951865 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:45.951830 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 20:16:45.952311 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:45.951885 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 20:16:46.952310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:46.952272 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 20:16:46.952738 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:46.952324 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 20:16:47.952396 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:47.952357 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 20:16:47.952803 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:47.952416 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 20:16:48.951590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:48.951553 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 20:16:48.951771 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:48.951632 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 20:16:49.952300 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:49.952258 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 20:16:49.952819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:49.952344 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 20:16:50.956363 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:50.956320 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:50.956818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:50.956799 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:50.957691 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:50.957665 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" Oct 31 20:16:54.202376 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.202343 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766"] Oct 31 20:16:54.205951 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.205934 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.258575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.258539 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766"] Oct 31 20:16:54.271587 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.271549 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd52h\" (UniqueName: \"kubernetes.io/projected/d629e1ff-ed7f-490a-ad03-947fa8f89dab-kube-api-access-bd52h\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.271750 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.271702 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.271799 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.271746 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.307252 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.307214 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:16:54.307426 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.307299 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:16:54.372909 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.372867 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd52h\" (UniqueName: \"kubernetes.io/projected/d629e1ff-ed7f-490a-ad03-947fa8f89dab-kube-api-access-bd52h\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.373147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.372978 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.373147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.373055 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.373725 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.373701 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.375325 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.375295 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.431024 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.430972 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd52h\" (UniqueName: \"kubernetes.io/projected/d629e1ff-ed7f-490a-ad03-947fa8f89dab-kube-api-access-bd52h\") pod \"maas-api-revoke-sub-2e1f5b5f-b7766\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.519501 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.519465 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:54.885159 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:16:54.885125 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd629e1ff_ed7f_490a_ad03_947fa8f89dab.slice/crio-40de23ed924bae38a09df5ab688687bbb291b4a3c2dac992929a8580de372fa0 WatchSource:0}: Error finding container 40de23ed924bae38a09df5ab688687bbb291b4a3c2dac992929a8580de372fa0: Status 404 returned error can't find the container with id 40de23ed924bae38a09df5ab688687bbb291b4a3c2dac992929a8580de372fa0 Oct 31 20:16:54.887293 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:54.887266 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766"] Oct 31 20:16:55.056401 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:55.056365 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" event={"ID":"d629e1ff-ed7f-490a-ad03-947fa8f89dab","Type":"ContainerStarted","Data":"40de23ed924bae38a09df5ab688687bbb291b4a3c2dac992929a8580de372fa0"} Oct 31 20:16:56.063485 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:56.063399 6481 generic.go:356] "Generic (PLEG): container finished" podID="d629e1ff-ed7f-490a-ad03-947fa8f89dab" containerID="037e96dd68b49bb89b6aeea69936ce08478ce62e6b70174f7b7f5c30d61ad547" exitCode=0 Oct 31 20:16:56.063485 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:56.063438 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" event={"ID":"d629e1ff-ed7f-490a-ad03-947fa8f89dab","Type":"ContainerDied","Data":"037e96dd68b49bb89b6aeea69936ce08478ce62e6b70174f7b7f5c30d61ad547"} Oct 31 20:16:57.196277 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.196253 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:57.305775 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.305741 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d629e1ff-ed7f-490a-ad03-947fa8f89dab-kube-api-access-bd52h\" (UniqueName: \"kubernetes.io/projected/d629e1ff-ed7f-490a-ad03-947fa8f89dab-kube-api-access-bd52h\") pod \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " Oct 31 20:16:57.305984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.305898 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-xks-ca\") pod \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " Oct 31 20:16:57.305984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.305957 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-openshift-ca\") pod \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\" (UID: \"d629e1ff-ed7f-490a-ad03-947fa8f89dab\") " Oct 31 20:16:57.306632 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.306601 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-openshift-ca" pod "d629e1ff-ed7f-490a-ad03-947fa8f89dab" (UID: "d629e1ff-ed7f-490a-ad03-947fa8f89dab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:16:57.310187 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.310146 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d629e1ff-ed7f-490a-ad03-947fa8f89dab-kube-api-access-bd52h" pod "d629e1ff-ed7f-490a-ad03-947fa8f89dab" (UID: "d629e1ff-ed7f-490a-ad03-947fa8f89dab"). InnerVolumeSpecName "kube-api-access-bd52h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:16:57.310339 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.310193 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-xks-ca" pod "d629e1ff-ed7f-490a-ad03-947fa8f89dab" (UID: "d629e1ff-ed7f-490a-ad03-947fa8f89dab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:16:57.328949 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:16:57.328872 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:16:57.407340 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.407308 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd52h\" (UniqueName: \"kubernetes.io/projected/d629e1ff-ed7f-490a-ad03-947fa8f89dab-kube-api-access-bd52h\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:16:57.407340 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.407336 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:16:57.407525 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:57.407348 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d629e1ff-ed7f-490a-ad03-947fa8f89dab-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:16:58.074619 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:58.074586 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" Oct 31 20:16:58.074619 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:58.074593 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766" event={"ID":"d629e1ff-ed7f-490a-ad03-947fa8f89dab","Type":"ContainerDied","Data":"40de23ed924bae38a09df5ab688687bbb291b4a3c2dac992929a8580de372fa0"} Oct 31 20:16:58.074849 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:16:58.074636 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40de23ed924bae38a09df5ab688687bbb291b4a3c2dac992929a8580de372fa0" Oct 31 20:17:06.606948 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.606916 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n"] Oct 31 20:17:06.607387 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.607373 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d629e1ff-ed7f-490a-ad03-947fa8f89dab" containerName="revoke-keys" Oct 31 20:17:06.607435 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.607388 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d629e1ff-ed7f-490a-ad03-947fa8f89dab" containerName="revoke-keys" Oct 31 20:17:06.607477 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.607465 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d629e1ff-ed7f-490a-ad03-947fa8f89dab" containerName="revoke-keys" Oct 31 20:17:06.610882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.610866 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.666835 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.666800 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n"] Oct 31 20:17:06.694393 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.694364 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68689370-f241-4573-9b44-781c098c1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.694565 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.694431 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbqx\" (UniqueName: \"kubernetes.io/projected/68689370-f241-4573-9b44-781c098c1720-kube-api-access-7fbqx\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.694565 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.694525 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68689370-f241-4573-9b44-781c098c1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.715209 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.715180 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:17:06.715209 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.715187 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:17:06.795445 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.795383 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68689370-f241-4573-9b44-781c098c1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.795631 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.795461 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbqx\" (UniqueName: \"kubernetes.io/projected/68689370-f241-4573-9b44-781c098c1720-kube-api-access-7fbqx\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.795631 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.795498 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68689370-f241-4573-9b44-781c098c1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.796275 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.796249 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68689370-f241-4573-9b44-781c098c1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.797771 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.797754 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68689370-f241-4573-9b44-781c098c1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.853673 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.853641 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbqx\" (UniqueName: \"kubernetes.io/projected/68689370-f241-4573-9b44-781c098c1720-kube-api-access-7fbqx\") pod \"maas-api-revoke-sub-ec843018-kjz5n\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:06.923592 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:06.923513 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:07.300198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:07.300116 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n"] Oct 31 20:17:08.125520 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:08.125490 6481 generic.go:356] "Generic (PLEG): container finished" podID="68689370-f241-4573-9b44-781c098c1720" containerID="1f6557e2010e0a833079344e6a0536d2280d419b1b87111e5e22c55a66fbf54d" exitCode=0 Oct 31 20:17:08.125882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:08.125571 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" event={"ID":"68689370-f241-4573-9b44-781c098c1720","Type":"ContainerDied","Data":"1f6557e2010e0a833079344e6a0536d2280d419b1b87111e5e22c55a66fbf54d"} Oct 31 20:17:08.125882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:08.125623 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" event={"ID":"68689370-f241-4573-9b44-781c098c1720","Type":"ContainerStarted","Data":"de409c4dafff27d2c2c33a71a1efb4eb6b72845f322eff3d3412eef5d6240214"} Oct 31 20:17:09.276904 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.276859 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:09.320416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.320378 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68689370-f241-4573-9b44-781c098c1720-kube-api-access-7fbqx\" (UniqueName: \"kubernetes.io/projected/68689370-f241-4573-9b44-781c098c1720-kube-api-access-7fbqx\") pod \"68689370-f241-4573-9b44-781c098c1720\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " Oct 31 20:17:09.320615 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.320459 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68689370-f241-4573-9b44-781c098c1720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68689370-f241-4573-9b44-781c098c1720-maas-api-openshift-ca\") pod \"68689370-f241-4573-9b44-781c098c1720\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " Oct 31 20:17:09.320615 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.320494 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68689370-f241-4573-9b44-781c098c1720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68689370-f241-4573-9b44-781c098c1720-maas-api-xks-ca\") pod \"68689370-f241-4573-9b44-781c098c1720\" (UID: \"68689370-f241-4573-9b44-781c098c1720\") " Oct 31 20:17:09.320930 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.320898 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68689370-f241-4573-9b44-781c098c1720-maas-api-openshift-ca" pod "68689370-f241-4573-9b44-781c098c1720" (UID: "68689370-f241-4573-9b44-781c098c1720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:17:09.323095 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.323060 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68689370-f241-4573-9b44-781c098c1720-maas-api-xks-ca" pod "68689370-f241-4573-9b44-781c098c1720" (UID: "68689370-f241-4573-9b44-781c098c1720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:17:09.323220 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.323169 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68689370-f241-4573-9b44-781c098c1720-kube-api-access-7fbqx" pod "68689370-f241-4573-9b44-781c098c1720" (UID: "68689370-f241-4573-9b44-781c098c1720"). InnerVolumeSpecName "kube-api-access-7fbqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:17:09.422291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.422214 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68689370-f241-4573-9b44-781c098c1720-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:09.422291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.422244 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68689370-f241-4573-9b44-781c098c1720-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:09.422291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:09.422259 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fbqx\" (UniqueName: \"kubernetes.io/projected/68689370-f241-4573-9b44-781c098c1720-kube-api-access-7fbqx\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:10.143443 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:10.143402 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" event={"ID":"68689370-f241-4573-9b44-781c098c1720","Type":"ContainerDied","Data":"de409c4dafff27d2c2c33a71a1efb4eb6b72845f322eff3d3412eef5d6240214"} Oct 31 20:17:10.143443 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:10.143446 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n" Oct 31 20:17:10.143730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:10.143491 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de409c4dafff27d2c2c33a71a1efb4eb6b72845f322eff3d3412eef5d6240214" Oct 31 20:17:27.324453 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:17:27.324373 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:17:41.740179 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.740140 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv"] Oct 31 20:17:41.740801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.740591 6481 cpu_manager.go:414] "containerMap: removing container" podUID="68689370-f241-4573-9b44-781c098c1720" containerName="revoke-keys" Oct 31 20:17:41.740801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.740603 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68689370-f241-4573-9b44-781c098c1720" containerName="revoke-keys" Oct 31 20:17:41.740801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.740666 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="68689370-f241-4573-9b44-781c098c1720" containerName="revoke-keys" Oct 31 20:17:41.743781 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.743756 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.783819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.783788 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhk6\" (UniqueName: \"kubernetes.io/projected/0bdd75e0-0416-4d87-b391-e820165036e7-kube-api-access-bfhk6\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.783959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.783832 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.783959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.783927 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.796154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.796123 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv"] Oct 31 20:17:41.844804 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.844774 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:17:41.844989 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.844951 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:17:41.885333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.885291 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhk6\" (UniqueName: \"kubernetes.io/projected/0bdd75e0-0416-4d87-b391-e820165036e7-kube-api-access-bfhk6\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.885333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.885337 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.885566 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.885461 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.886176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.886154 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.887899 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.887860 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:41.943559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:41.943529 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhk6\" (UniqueName: \"kubernetes.io/projected/0bdd75e0-0416-4d87-b391-e820165036e7-kube-api-access-bfhk6\") pod \"maas-api-revoke-sub-046e1343-5c6rv\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:42.056886 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:42.056820 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:42.429215 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:17:42.429183 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdd75e0_0416_4d87_b391_e820165036e7.slice/crio-3269cd31764d1e080042271240e3c834bcbf34586755457db87cb4876fe1ecf5 WatchSource:0}: Error finding container 3269cd31764d1e080042271240e3c834bcbf34586755457db87cb4876fe1ecf5: Status 404 returned error can't find the container with id 3269cd31764d1e080042271240e3c834bcbf34586755457db87cb4876fe1ecf5 Oct 31 20:17:42.432629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:42.432606 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv"] Oct 31 20:17:43.326093 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:43.326010 6481 generic.go:356] "Generic (PLEG): container finished" podID="0bdd75e0-0416-4d87-b391-e820165036e7" containerID="b4d6effe025e7ac8e6b068211c1fcb3e3e0782003f36402076d27350c99a296c" exitCode=0 Oct 31 20:17:43.326497 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:43.326092 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" event={"ID":"0bdd75e0-0416-4d87-b391-e820165036e7","Type":"ContainerDied","Data":"b4d6effe025e7ac8e6b068211c1fcb3e3e0782003f36402076d27350c99a296c"} Oct 31 20:17:43.326497 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:43.326187 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" event={"ID":"0bdd75e0-0416-4d87-b391-e820165036e7","Type":"ContainerStarted","Data":"3269cd31764d1e080042271240e3c834bcbf34586755457db87cb4876fe1ecf5"} Oct 31 20:17:44.452440 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.452414 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:44.509203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.509172 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-xks-ca\") pod \"0bdd75e0-0416-4d87-b391-e820165036e7\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " Oct 31 20:17:44.509203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.509208 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdd75e0-0416-4d87-b391-e820165036e7-kube-api-access-bfhk6\" (UniqueName: \"kubernetes.io/projected/0bdd75e0-0416-4d87-b391-e820165036e7-kube-api-access-bfhk6\") pod \"0bdd75e0-0416-4d87-b391-e820165036e7\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " Oct 31 20:17:44.509404 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.509262 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-openshift-ca\") pod \"0bdd75e0-0416-4d87-b391-e820165036e7\" (UID: \"0bdd75e0-0416-4d87-b391-e820165036e7\") " Oct 31 20:17:44.509704 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.509681 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-openshift-ca" pod "0bdd75e0-0416-4d87-b391-e820165036e7" (UID: "0bdd75e0-0416-4d87-b391-e820165036e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:17:44.511310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.511283 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-xks-ca" pod "0bdd75e0-0416-4d87-b391-e820165036e7" (UID: "0bdd75e0-0416-4d87-b391-e820165036e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:17:44.511507 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.511490 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdd75e0-0416-4d87-b391-e820165036e7-kube-api-access-bfhk6" pod "0bdd75e0-0416-4d87-b391-e820165036e7" (UID: "0bdd75e0-0416-4d87-b391-e820165036e7"). InnerVolumeSpecName "kube-api-access-bfhk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:17:44.610629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.610555 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:44.610629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.610584 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdd75e0-0416-4d87-b391-e820165036e7-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:44.610629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:44.610595 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfhk6\" (UniqueName: \"kubernetes.io/projected/0bdd75e0-0416-4d87-b391-e820165036e7-kube-api-access-bfhk6\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:45.337070 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:45.337038 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" Oct 31 20:17:45.337070 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:45.337048 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv" event={"ID":"0bdd75e0-0416-4d87-b391-e820165036e7","Type":"ContainerDied","Data":"3269cd31764d1e080042271240e3c834bcbf34586755457db87cb4876fe1ecf5"} Oct 31 20:17:45.337287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:45.337096 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3269cd31764d1e080042271240e3c834bcbf34586755457db87cb4876fe1ecf5" Oct 31 20:17:52.496893 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.496855 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg"] Oct 31 20:17:52.497565 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.497544 6481 cpu_manager.go:414] "containerMap: removing container" podUID="0bdd75e0-0416-4d87-b391-e820165036e7" containerName="revoke-keys" Oct 31 20:17:52.497616 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.497570 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdd75e0-0416-4d87-b391-e820165036e7" containerName="revoke-keys" Oct 31 20:17:52.497731 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.497718 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdd75e0-0416-4d87-b391-e820165036e7" containerName="revoke-keys" Oct 31 20:17:52.501497 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.501476 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.552908 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.552879 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg"] Oct 31 20:17:52.588448 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.588418 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.588610 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.588454 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.588610 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.588541 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzlk\" (UniqueName: \"kubernetes.io/projected/ddb01ad0-881c-4567-858e-4d9be951ebae-kube-api-access-6jzlk\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.602143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.602117 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:17:52.602310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.602294 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:17:52.690007 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.689972 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzlk\" (UniqueName: \"kubernetes.io/projected/ddb01ad0-881c-4567-858e-4d9be951ebae-kube-api-access-6jzlk\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.690186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.690154 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.690256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.690184 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.690816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.690794 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.692582 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.692563 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.747920 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.747856 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzlk\" (UniqueName: \"kubernetes.io/projected/ddb01ad0-881c-4567-858e-4d9be951ebae-kube-api-access-6jzlk\") pod \"maas-api-revoke-sub-d3fba056-9p5dg\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:52.814837 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:52.814809 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:53.183221 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:17:53.183181 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb01ad0_881c_4567_858e_4d9be951ebae.slice/crio-b83b53da19ffac4cc4ce99056dfa769e6023a2b66a40ab570ff9738dca7af21c WatchSource:0}: Error finding container b83b53da19ffac4cc4ce99056dfa769e6023a2b66a40ab570ff9738dca7af21c: Status 404 returned error can't find the container with id b83b53da19ffac4cc4ce99056dfa769e6023a2b66a40ab570ff9738dca7af21c Oct 31 20:17:53.185616 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:53.185585 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg"] Oct 31 20:17:53.377311 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:53.377281 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" event={"ID":"ddb01ad0-881c-4567-858e-4d9be951ebae","Type":"ContainerStarted","Data":"b83b53da19ffac4cc4ce99056dfa769e6023a2b66a40ab570ff9738dca7af21c"} Oct 31 20:17:54.383486 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:54.383387 6481 generic.go:356] "Generic (PLEG): container finished" podID="ddb01ad0-881c-4567-858e-4d9be951ebae" containerID="9f5f2c8263ea9aa854190dd2b2902dc1ad275c3f40c16bf49cd27317fc08b27d" exitCode=0 Oct 31 20:17:54.383917 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:54.383475 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" event={"ID":"ddb01ad0-881c-4567-858e-4d9be951ebae","Type":"ContainerDied","Data":"9f5f2c8263ea9aa854190dd2b2902dc1ad275c3f40c16bf49cd27317fc08b27d"} Oct 31 20:17:55.579398 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.579359 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:55.721303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.721258 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-openshift-ca\") pod \"ddb01ad0-881c-4567-858e-4d9be951ebae\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " Oct 31 20:17:55.721529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.721408 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb01ad0-881c-4567-858e-4d9be951ebae-kube-api-access-6jzlk\" (UniqueName: \"kubernetes.io/projected/ddb01ad0-881c-4567-858e-4d9be951ebae-kube-api-access-6jzlk\") pod \"ddb01ad0-881c-4567-858e-4d9be951ebae\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " Oct 31 20:17:55.721529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.721484 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-xks-ca\") pod \"ddb01ad0-881c-4567-858e-4d9be951ebae\" (UID: \"ddb01ad0-881c-4567-858e-4d9be951ebae\") " Oct 31 20:17:55.723079 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.723048 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-openshift-ca" pod "ddb01ad0-881c-4567-858e-4d9be951ebae" (UID: "ddb01ad0-881c-4567-858e-4d9be951ebae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:17:55.724765 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.724726 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-xks-ca" pod "ddb01ad0-881c-4567-858e-4d9be951ebae" (UID: "ddb01ad0-881c-4567-858e-4d9be951ebae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:17:55.822208 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.822164 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb01ad0-881c-4567-858e-4d9be951ebae-kube-api-access-6jzlk" pod "ddb01ad0-881c-4567-858e-4d9be951ebae" (UID: "ddb01ad0-881c-4567-858e-4d9be951ebae"). InnerVolumeSpecName "kube-api-access-6jzlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:17:55.828016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.827317 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:55.828016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.827349 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb01ad0-881c-4567-858e-4d9be951ebae-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:55.828016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:55.827367 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzlk\" (UniqueName: \"kubernetes.io/projected/ddb01ad0-881c-4567-858e-4d9be951ebae-kube-api-access-6jzlk\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:17:56.424649 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:56.424601 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" event={"ID":"ddb01ad0-881c-4567-858e-4d9be951ebae","Type":"ContainerDied","Data":"b83b53da19ffac4cc4ce99056dfa769e6023a2b66a40ab570ff9738dca7af21c"} Oct 31 20:17:56.424952 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:56.424924 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b83b53da19ffac4cc4ce99056dfa769e6023a2b66a40ab570ff9738dca7af21c" Oct 31 20:17:56.425194 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:17:56.425179 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg" Oct 31 20:17:57.366923 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:17:57.366848 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:18:27.282250 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.282205 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n"] Oct 31 20:18:27.283190 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.283163 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ddb01ad0-881c-4567-858e-4d9be951ebae" containerName="revoke-keys" Oct 31 20:18:27.283190 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.283191 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb01ad0-881c-4567-858e-4d9be951ebae" containerName="revoke-keys" Oct 31 20:18:27.283425 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.283411 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb01ad0-881c-4567-858e-4d9be951ebae" containerName="revoke-keys" Oct 31 20:18:27.288437 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.288410 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.289230 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.289202 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.289352 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.289327 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgt6\" (UniqueName: \"kubernetes.io/projected/d138e030-c19b-473e-a17e-86753acb09e3-kube-api-access-4xgt6\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.289422 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.289388 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.324356 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:18:27.324320 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:18:27.336832 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.336802 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n"] Oct 31 20:18:27.385963 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.385926 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:18:27.385963 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.385942 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:18:27.390448 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.390419 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.390575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.390506 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgt6\" (UniqueName: \"kubernetes.io/projected/d138e030-c19b-473e-a17e-86753acb09e3-kube-api-access-4xgt6\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.390575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.390555 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.391215 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.391191 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.392739 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.392720 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.448742 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.448713 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgt6\" (UniqueName: \"kubernetes.io/projected/d138e030-c19b-473e-a17e-86753acb09e3-kube-api-access-4xgt6\") pod \"maas-api-revoke-sub-ae594979-s4x5n\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.601944 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.601862 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:27.967970 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:18:27.967935 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd138e030_c19b_473e_a17e_86753acb09e3.slice/crio-0b24c90a43d87b2465803549f3b0ea09641dab33b00ae45961cfde59bc3cbf9e WatchSource:0}: Error finding container 0b24c90a43d87b2465803549f3b0ea09641dab33b00ae45961cfde59bc3cbf9e: Status 404 returned error can't find the container with id 0b24c90a43d87b2465803549f3b0ea09641dab33b00ae45961cfde59bc3cbf9e Oct 31 20:18:27.970246 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:27.970221 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n"] Oct 31 20:18:28.630018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:28.629920 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" event={"ID":"d138e030-c19b-473e-a17e-86753acb09e3","Type":"ContainerStarted","Data":"0abe854beef16f5b01155e0f08a3add76695fcb4955329a4c2c33d919ad8dd61"} Oct 31 20:18:28.630018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:28.629973 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" event={"ID":"d138e030-c19b-473e-a17e-86753acb09e3","Type":"ContainerStarted","Data":"0b24c90a43d87b2465803549f3b0ea09641dab33b00ae45961cfde59bc3cbf9e"} Oct 31 20:18:28.745412 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:28.745364 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" podStartSLOduration=1.7453503609999999 podStartE2EDuration="1.745350361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:18:28.744716412 +0000 UTC m=+3679.984939555" watchObservedRunningTime="2026-10-31 20:18:28.745350361 +0000 UTC m=+3679.985573491" Oct 31 20:18:29.635662 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:29.635630 6481 generic.go:356] "Generic (PLEG): container finished" podID="d138e030-c19b-473e-a17e-86753acb09e3" containerID="0abe854beef16f5b01155e0f08a3add76695fcb4955329a4c2c33d919ad8dd61" exitCode=0 Oct 31 20:18:29.636102 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:29.635696 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" event={"ID":"d138e030-c19b-473e-a17e-86753acb09e3","Type":"ContainerDied","Data":"0abe854beef16f5b01155e0f08a3add76695fcb4955329a4c2c33d919ad8dd61"} Oct 31 20:18:30.768714 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.768687 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:30.818780 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.818750 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d138e030-c19b-473e-a17e-86753acb09e3-kube-api-access-4xgt6\" (UniqueName: \"kubernetes.io/projected/d138e030-c19b-473e-a17e-86753acb09e3-kube-api-access-4xgt6\") pod \"d138e030-c19b-473e-a17e-86753acb09e3\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " Oct 31 20:18:30.818938 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.818825 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-xks-ca\") pod \"d138e030-c19b-473e-a17e-86753acb09e3\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " Oct 31 20:18:30.818938 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.818864 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-openshift-ca\") pod \"d138e030-c19b-473e-a17e-86753acb09e3\" (UID: \"d138e030-c19b-473e-a17e-86753acb09e3\") " Oct 31 20:18:30.819279 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.819253 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-openshift-ca" pod "d138e030-c19b-473e-a17e-86753acb09e3" (UID: "d138e030-c19b-473e-a17e-86753acb09e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:18:30.820859 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.820837 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-xks-ca" pod "d138e030-c19b-473e-a17e-86753acb09e3" (UID: "d138e030-c19b-473e-a17e-86753acb09e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:18:30.820959 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.820867 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d138e030-c19b-473e-a17e-86753acb09e3-kube-api-access-4xgt6" pod "d138e030-c19b-473e-a17e-86753acb09e3" (UID: "d138e030-c19b-473e-a17e-86753acb09e3"). InnerVolumeSpecName "kube-api-access-4xgt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:18:30.919882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.919806 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xgt6\" (UniqueName: \"kubernetes.io/projected/d138e030-c19b-473e-a17e-86753acb09e3-kube-api-access-4xgt6\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:18:30.919882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.919834 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:18:30.919882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:30.919848 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d138e030-c19b-473e-a17e-86753acb09e3-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:18:31.645947 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:31.645917 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" Oct 31 20:18:31.646143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:31.645916 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n" event={"ID":"d138e030-c19b-473e-a17e-86753acb09e3","Type":"ContainerDied","Data":"0b24c90a43d87b2465803549f3b0ea09641dab33b00ae45961cfde59bc3cbf9e"} Oct 31 20:18:31.646143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:31.646048 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b24c90a43d87b2465803549f3b0ea09641dab33b00ae45961cfde59bc3cbf9e" Oct 31 20:18:55.227448 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.227372 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd"] Oct 31 20:18:55.227890 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.227776 6481 cpu_manager.go:414] "containerMap: removing container" podUID="d138e030-c19b-473e-a17e-86753acb09e3" containerName="revoke-keys" Oct 31 20:18:55.227890 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.227786 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d138e030-c19b-473e-a17e-86753acb09e3" containerName="revoke-keys" Oct 31 20:18:55.227890 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.227853 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d138e030-c19b-473e-a17e-86753acb09e3" containerName="revoke-keys" Oct 31 20:18:55.230864 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.230846 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.275492 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.275456 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5jd\" (UniqueName: \"kubernetes.io/projected/c7f03823-eaca-4724-9b99-19855a1079a8-kube-api-access-lh5jd\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.275760 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.275740 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.276146 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.275975 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.283627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.283605 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd"] Oct 31 20:18:55.332145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.332102 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:18:55.332145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.332145 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:18:55.377517 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.377483 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.377679 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.377573 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5jd\" (UniqueName: \"kubernetes.io/projected/c7f03823-eaca-4724-9b99-19855a1079a8-kube-api-access-lh5jd\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.377679 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.377634 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.378306 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.378284 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.379946 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.379929 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.437717 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.437688 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5jd\" (UniqueName: \"kubernetes.io/projected/c7f03823-eaca-4724-9b99-19855a1079a8-kube-api-access-lh5jd\") pod \"maas-api-revoke-sub-a4aef7d7-26mxd\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.543408 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.543319 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:55.910084 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:18:55.910052 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f03823_eaca_4724_9b99_19855a1079a8.slice/crio-b13d57776caec6cca2200c6940046a5f495a2b670253aa0a38137233fe930c33 WatchSource:0}: Error finding container b13d57776caec6cca2200c6940046a5f495a2b670253aa0a38137233fe930c33: Status 404 returned error can't find the container with id b13d57776caec6cca2200c6940046a5f495a2b670253aa0a38137233fe930c33 Oct 31 20:18:55.913088 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:55.913067 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd"] Oct 31 20:18:56.810409 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:56.810377 6481 generic.go:356] "Generic (PLEG): container finished" podID="c7f03823-eaca-4724-9b99-19855a1079a8" containerID="cdc64010a76afbb0c8b94877a40ad9d8c5d73d0ec5ec9062de2d06f83fe41ddf" exitCode=0 Oct 31 20:18:56.810851 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:56.810471 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" event={"ID":"c7f03823-eaca-4724-9b99-19855a1079a8","Type":"ContainerDied","Data":"cdc64010a76afbb0c8b94877a40ad9d8c5d73d0ec5ec9062de2d06f83fe41ddf"} Oct 31 20:18:56.810851 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:56.810543 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" event={"ID":"c7f03823-eaca-4724-9b99-19855a1079a8","Type":"ContainerStarted","Data":"b13d57776caec6cca2200c6940046a5f495a2b670253aa0a38137233fe930c33"} Oct 31 20:18:57.325090 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:18:57.325051 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:18:57.948125 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:57.948094 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:58.004727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.004688 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f03823-eaca-4724-9b99-19855a1079a8-kube-api-access-lh5jd\" (UniqueName: \"kubernetes.io/projected/c7f03823-eaca-4724-9b99-19855a1079a8-kube-api-access-lh5jd\") pod \"c7f03823-eaca-4724-9b99-19855a1079a8\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " Oct 31 20:18:58.004727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.004726 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-xks-ca\") pod \"c7f03823-eaca-4724-9b99-19855a1079a8\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " Oct 31 20:18:58.004989 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.004783 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-openshift-ca\") pod \"c7f03823-eaca-4724-9b99-19855a1079a8\" (UID: \"c7f03823-eaca-4724-9b99-19855a1079a8\") " Oct 31 20:18:58.005260 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.005231 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-openshift-ca" pod "c7f03823-eaca-4724-9b99-19855a1079a8" (UID: "c7f03823-eaca-4724-9b99-19855a1079a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:18:58.006981 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.006955 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-xks-ca" pod "c7f03823-eaca-4724-9b99-19855a1079a8" (UID: "c7f03823-eaca-4724-9b99-19855a1079a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:18:58.007111 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.006988 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f03823-eaca-4724-9b99-19855a1079a8-kube-api-access-lh5jd" pod "c7f03823-eaca-4724-9b99-19855a1079a8" (UID: "c7f03823-eaca-4724-9b99-19855a1079a8"). InnerVolumeSpecName "kube-api-access-lh5jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:18:58.106472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.106399 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5jd\" (UniqueName: \"kubernetes.io/projected/c7f03823-eaca-4724-9b99-19855a1079a8-kube-api-access-lh5jd\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:18:58.106472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.106425 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:18:58.106472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.106438 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f03823-eaca-4724-9b99-19855a1079a8-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:18:58.821314 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.821283 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" Oct 31 20:18:58.821498 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.821319 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd" event={"ID":"c7f03823-eaca-4724-9b99-19855a1079a8","Type":"ContainerDied","Data":"b13d57776caec6cca2200c6940046a5f495a2b670253aa0a38137233fe930c33"} Oct 31 20:18:58.821498 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:18:58.821370 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b13d57776caec6cca2200c6940046a5f495a2b670253aa0a38137233fe930c33" Oct 31 20:19:24.050343 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:24.050308 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j"] Oct 31 20:19:24.101702 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:24.101661 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4aa22f8-sj85j"] Oct 31 20:19:24.878561 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:24.878525 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0800e2-e40a-49d6-acd2-db082b08c0ec" path="/var/lib/kubelet/pods/3b0800e2-e40a-49d6-acd2-db082b08c0ec/volumes" Oct 31 20:19:25.713938 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.713899 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv"] Oct 31 20:19:25.714354 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.714332 6481 cpu_manager.go:414] "containerMap: removing container" podUID="c7f03823-eaca-4724-9b99-19855a1079a8" containerName="revoke-keys" Oct 31 20:19:25.714354 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.714344 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f03823-eaca-4724-9b99-19855a1079a8" containerName="revoke-keys" Oct 31 20:19:25.714433 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.714406 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f03823-eaca-4724-9b99-19855a1079a8" containerName="revoke-keys" Oct 31 20:19:25.717540 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.717522 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.771153 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.771120 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv"] Oct 31 20:19:25.819980 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.819936 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:19:25.819980 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.819982 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:19:25.845417 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.845379 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.845625 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.845464 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwhd\" (UniqueName: \"kubernetes.io/projected/8dbf319b-bab3-4716-8dbe-0006c5f32af8-kube-api-access-xlwhd\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.845625 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.845496 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.947167 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.947118 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwhd\" (UniqueName: \"kubernetes.io/projected/8dbf319b-bab3-4716-8dbe-0006c5f32af8-kube-api-access-xlwhd\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.947167 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.947169 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.947434 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.947307 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.947982 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.947959 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:25.949728 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:25.949697 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:26.005459 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:26.005388 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwhd\" (UniqueName: \"kubernetes.io/projected/8dbf319b-bab3-4716-8dbe-0006c5f32af8-kube-api-access-xlwhd\") pod \"maas-api-revoke-sub-79805ca9-v56lv\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:26.030083 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:26.030036 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:26.395552 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:19:26.395523 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbf319b_bab3_4716_8dbe_0006c5f32af8.slice/crio-020871c9a0a8e97a96e4a9c90ec5079b09059075ed9e963eaad67e4778a39f5f WatchSource:0}: Error finding container 020871c9a0a8e97a96e4a9c90ec5079b09059075ed9e963eaad67e4778a39f5f: Status 404 returned error can't find the container with id 020871c9a0a8e97a96e4a9c90ec5079b09059075ed9e963eaad67e4778a39f5f Oct 31 20:19:26.399200 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:26.399176 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv"] Oct 31 20:19:26.990219 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:26.990176 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" event={"ID":"8dbf319b-bab3-4716-8dbe-0006c5f32af8","Type":"ContainerStarted","Data":"fdd55b6db20546f6bf9c35f12d0b32147fc7b0083ec5336163406004a8441f00"} Oct 31 20:19:26.990635 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:26.990244 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" event={"ID":"8dbf319b-bab3-4716-8dbe-0006c5f32af8","Type":"ContainerStarted","Data":"020871c9a0a8e97a96e4a9c90ec5079b09059075ed9e963eaad67e4778a39f5f"} Oct 31 20:19:27.107399 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:27.107339 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" podStartSLOduration=2.107326251 podStartE2EDuration="2.107326251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:27.105861485 +0000 UTC m=+3738.346084615" watchObservedRunningTime="2026-10-31 20:19:27.107326251 +0000 UTC m=+3738.347549380" Oct 31 20:19:27.324769 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:19:27.324690 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:19:27.996535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:27.996497 6481 generic.go:356] "Generic (PLEG): container finished" podID="8dbf319b-bab3-4716-8dbe-0006c5f32af8" containerID="fdd55b6db20546f6bf9c35f12d0b32147fc7b0083ec5336163406004a8441f00" exitCode=0 Oct 31 20:19:27.996985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:27.996567 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" event={"ID":"8dbf319b-bab3-4716-8dbe-0006c5f32af8","Type":"ContainerDied","Data":"fdd55b6db20546f6bf9c35f12d0b32147fc7b0083ec5336163406004a8441f00"} Oct 31 20:19:29.135391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.135358 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:29.176405 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.176365 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-openshift-ca\") pod \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " Oct 31 20:19:29.176590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.176436 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbf319b-bab3-4716-8dbe-0006c5f32af8-kube-api-access-xlwhd\" (UniqueName: \"kubernetes.io/projected/8dbf319b-bab3-4716-8dbe-0006c5f32af8-kube-api-access-xlwhd\") pod \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " Oct 31 20:19:29.176590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.176544 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-xks-ca\") pod \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\" (UID: \"8dbf319b-bab3-4716-8dbe-0006c5f32af8\") " Oct 31 20:19:29.176792 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.176769 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-openshift-ca" pod "8dbf319b-bab3-4716-8dbe-0006c5f32af8" (UID: "8dbf319b-bab3-4716-8dbe-0006c5f32af8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:19:29.178714 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.178687 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-xks-ca" pod "8dbf319b-bab3-4716-8dbe-0006c5f32af8" (UID: "8dbf319b-bab3-4716-8dbe-0006c5f32af8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:19:29.178714 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.178698 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbf319b-bab3-4716-8dbe-0006c5f32af8-kube-api-access-xlwhd" pod "8dbf319b-bab3-4716-8dbe-0006c5f32af8" (UID: "8dbf319b-bab3-4716-8dbe-0006c5f32af8"). InnerVolumeSpecName "kube-api-access-xlwhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:19:29.277869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.277774 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:29.277869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.277806 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dbf319b-bab3-4716-8dbe-0006c5f32af8-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:29.277869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:29.277817 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlwhd\" (UniqueName: \"kubernetes.io/projected/8dbf319b-bab3-4716-8dbe-0006c5f32af8-kube-api-access-xlwhd\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:30.008039 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:30.007990 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" Oct 31 20:19:30.008039 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:30.008017 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv" event={"ID":"8dbf319b-bab3-4716-8dbe-0006c5f32af8","Type":"ContainerDied","Data":"020871c9a0a8e97a96e4a9c90ec5079b09059075ed9e963eaad67e4778a39f5f"} Oct 31 20:19:30.008354 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:30.008077 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="020871c9a0a8e97a96e4a9c90ec5079b09059075ed9e963eaad67e4778a39f5f" Oct 31 20:19:45.560159 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:45.560102 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:19:45.560734 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:45.560192 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:19:45.777018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:45.775141 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:19:45.777018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:45.775227 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:19:52.602583 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.602551 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm"] Oct 31 20:19:52.602989 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.602972 6481 cpu_manager.go:414] "containerMap: removing container" podUID="8dbf319b-bab3-4716-8dbe-0006c5f32af8" containerName="revoke-keys" Oct 31 20:19:52.603056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.603003 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbf319b-bab3-4716-8dbe-0006c5f32af8" containerName="revoke-keys" Oct 31 20:19:52.603111 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.603099 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbf319b-bab3-4716-8dbe-0006c5f32af8" containerName="revoke-keys" Oct 31 20:19:52.606286 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.606257 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.659747 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.659713 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm"] Oct 31 20:19:52.708341 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.708316 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:19:52.708488 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.708387 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:19:52.753831 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.753803 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54409742-cf47-436b-ad02-6827776112a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.753967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.753919 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54409742-cf47-436b-ad02-6827776112a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.753967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.753955 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xm5t\" (UniqueName: \"kubernetes.io/projected/54409742-cf47-436b-ad02-6827776112a6-kube-api-access-5xm5t\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.855271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.855199 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54409742-cf47-436b-ad02-6827776112a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.855271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.855246 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xm5t\" (UniqueName: \"kubernetes.io/projected/54409742-cf47-436b-ad02-6827776112a6-kube-api-access-5xm5t\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.855495 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.855300 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54409742-cf47-436b-ad02-6827776112a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.855888 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.855863 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54409742-cf47-436b-ad02-6827776112a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.857434 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.857415 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54409742-cf47-436b-ad02-6827776112a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.913055 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.913027 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xm5t\" (UniqueName: \"kubernetes.io/projected/54409742-cf47-436b-ad02-6827776112a6-kube-api-access-5xm5t\") pod \"maas-api-revoke-sub-0b7e8edb-6xdkm\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:52.918852 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:52.918826 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:53.281662 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:19:53.281630 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54409742_cf47_436b_ad02_6827776112a6.slice/crio-8f0eaed7c9ab40c27da515482e67d70c5396cd1135f6555082b573b2fcd0550c WatchSource:0}: Error finding container 8f0eaed7c9ab40c27da515482e67d70c5396cd1135f6555082b573b2fcd0550c: Status 404 returned error can't find the container with id 8f0eaed7c9ab40c27da515482e67d70c5396cd1135f6555082b573b2fcd0550c Oct 31 20:19:53.284549 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:53.284527 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm"] Oct 31 20:19:54.147391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:54.147357 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" event={"ID":"54409742-cf47-436b-ad02-6827776112a6","Type":"ContainerStarted","Data":"14f44f9b350d53da8663965490a02ebc867a7bfc9727edb268f722a02650f64c"} Oct 31 20:19:54.147739 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:54.147418 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" event={"ID":"54409742-cf47-436b-ad02-6827776112a6","Type":"ContainerStarted","Data":"8f0eaed7c9ab40c27da515482e67d70c5396cd1135f6555082b573b2fcd0550c"} Oct 31 20:19:54.264093 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:54.264046 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" podStartSLOduration=2.264033092 podStartE2EDuration="2.264033092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:54.263536314 +0000 UTC m=+3765.503759445" watchObservedRunningTime="2026-10-31 20:19:54.264033092 +0000 UTC m=+3765.504256221" Oct 31 20:19:55.153870 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:55.153829 6481 generic.go:356] "Generic (PLEG): container finished" podID="54409742-cf47-436b-ad02-6827776112a6" containerID="14f44f9b350d53da8663965490a02ebc867a7bfc9727edb268f722a02650f64c" exitCode=0 Oct 31 20:19:55.154287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:55.153891 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" event={"ID":"54409742-cf47-436b-ad02-6827776112a6","Type":"ContainerDied","Data":"14f44f9b350d53da8663965490a02ebc867a7bfc9727edb268f722a02650f64c"} Oct 31 20:19:56.302041 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.302011 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:56.393449 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.393422 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54409742-cf47-436b-ad02-6827776112a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54409742-cf47-436b-ad02-6827776112a6-maas-api-xks-ca\") pod \"54409742-cf47-436b-ad02-6827776112a6\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " Oct 31 20:19:56.393612 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.393465 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54409742-cf47-436b-ad02-6827776112a6-kube-api-access-5xm5t\" (UniqueName: \"kubernetes.io/projected/54409742-cf47-436b-ad02-6827776112a6-kube-api-access-5xm5t\") pod \"54409742-cf47-436b-ad02-6827776112a6\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " Oct 31 20:19:56.393612 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.393520 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54409742-cf47-436b-ad02-6827776112a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54409742-cf47-436b-ad02-6827776112a6-maas-api-openshift-ca\") pod \"54409742-cf47-436b-ad02-6827776112a6\" (UID: \"54409742-cf47-436b-ad02-6827776112a6\") " Oct 31 20:19:56.393937 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.393911 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54409742-cf47-436b-ad02-6827776112a6-maas-api-openshift-ca" pod "54409742-cf47-436b-ad02-6827776112a6" (UID: "54409742-cf47-436b-ad02-6827776112a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:19:56.395442 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.395417 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54409742-cf47-436b-ad02-6827776112a6-maas-api-xks-ca" pod "54409742-cf47-436b-ad02-6827776112a6" (UID: "54409742-cf47-436b-ad02-6827776112a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:19:56.395538 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.395484 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54409742-cf47-436b-ad02-6827776112a6-kube-api-access-5xm5t" pod "54409742-cf47-436b-ad02-6827776112a6" (UID: "54409742-cf47-436b-ad02-6827776112a6"). InnerVolumeSpecName "kube-api-access-5xm5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:19:56.494204 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.494175 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54409742-cf47-436b-ad02-6827776112a6-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:56.494204 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.494203 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xm5t\" (UniqueName: \"kubernetes.io/projected/54409742-cf47-436b-ad02-6827776112a6-kube-api-access-5xm5t\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:56.494391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:56.494218 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54409742-cf47-436b-ad02-6827776112a6-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:57.163960 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:57.163880 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" Oct 31 20:19:57.163960 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:57.163919 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm" event={"ID":"54409742-cf47-436b-ad02-6827776112a6","Type":"ContainerDied","Data":"8f0eaed7c9ab40c27da515482e67d70c5396cd1135f6555082b573b2fcd0550c"} Oct 31 20:19:57.164242 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:19:57.163973 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f0eaed7c9ab40c27da515482e67d70c5396cd1135f6555082b573b2fcd0550c" Oct 31 20:19:57.323599 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:19:57.323564 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:20:03.992164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:03.992124 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt"] Oct 31 20:20:03.992823 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:03.992797 6481 cpu_manager.go:414] "containerMap: removing container" podUID="54409742-cf47-436b-ad02-6827776112a6" containerName="revoke-keys" Oct 31 20:20:03.992912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:03.992826 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54409742-cf47-436b-ad02-6827776112a6" containerName="revoke-keys" Oct 31 20:20:03.993062 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:03.993043 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="54409742-cf47-436b-ad02-6827776112a6" containerName="revoke-keys" Oct 31 20:20:03.996797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:03.996770 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.048203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.048170 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt"] Oct 31 20:20:04.058700 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.058664 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.058700 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.058703 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.058957 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.058804 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9ld\" (UniqueName: \"kubernetes.io/projected/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-kube-api-access-nj9ld\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.096649 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.096607 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:20:04.096837 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.096702 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:20:04.159399 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.159361 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.159399 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.159403 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.159611 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.159446 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9ld\" (UniqueName: \"kubernetes.io/projected/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-kube-api-access-nj9ld\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.160085 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.160065 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.161783 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.161760 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.217927 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.217894 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9ld\" (UniqueName: \"kubernetes.io/projected/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-kube-api-access-nj9ld\") pod \"maas-api-revoke-sub-3d4156f1-ptwvt\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.311435 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.311346 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:04.676175 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:20:04.676143 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bee5e7_e3ca_44ac_b9f2_6d7797184f55.slice/crio-b300c8497ad528fcb76a58b1307b1a46adc4e0488ba626df2ed603d930bf949f WatchSource:0}: Error finding container b300c8497ad528fcb76a58b1307b1a46adc4e0488ba626df2ed603d930bf949f: Status 404 returned error can't find the container with id b300c8497ad528fcb76a58b1307b1a46adc4e0488ba626df2ed603d930bf949f Oct 31 20:20:04.678947 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:04.678926 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt"] Oct 31 20:20:05.204859 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:05.204817 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" event={"ID":"56bee5e7-e3ca-44ac-b9f2-6d7797184f55","Type":"ContainerStarted","Data":"1e03b7f9ef29b217ca79229fd4967984782bde84fabf158234323ebdc7f30f1e"} Oct 31 20:20:05.205256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:05.204885 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" event={"ID":"56bee5e7-e3ca-44ac-b9f2-6d7797184f55","Type":"ContainerStarted","Data":"b300c8497ad528fcb76a58b1307b1a46adc4e0488ba626df2ed603d930bf949f"} Oct 31 20:20:05.320304 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:05.320202 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" podStartSLOduration=2.320187186 podStartE2EDuration="2.320187186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:20:05.318546812 +0000 UTC m=+3776.558769941" watchObservedRunningTime="2026-10-31 20:20:05.320187186 +0000 UTC m=+3776.560410348" Oct 31 20:20:06.214603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:06.214569 6481 generic.go:356] "Generic (PLEG): container finished" podID="56bee5e7-e3ca-44ac-b9f2-6d7797184f55" containerID="1e03b7f9ef29b217ca79229fd4967984782bde84fabf158234323ebdc7f30f1e" exitCode=0 Oct 31 20:20:06.215052 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:06.214672 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" event={"ID":"56bee5e7-e3ca-44ac-b9f2-6d7797184f55","Type":"ContainerDied","Data":"1e03b7f9ef29b217ca79229fd4967984782bde84fabf158234323ebdc7f30f1e"} Oct 31 20:20:07.355352 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.355319 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:07.392486 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.392455 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-kube-api-access-nj9ld\" (UniqueName: \"kubernetes.io/projected/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-kube-api-access-nj9ld\") pod \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " Oct 31 20:20:07.392624 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.392517 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-xks-ca\") pod \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " Oct 31 20:20:07.392666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.392617 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-openshift-ca\") pod \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\" (UID: \"56bee5e7-e3ca-44ac-b9f2-6d7797184f55\") " Oct 31 20:20:07.393035 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.392982 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-openshift-ca" pod "56bee5e7-e3ca-44ac-b9f2-6d7797184f55" (UID: "56bee5e7-e3ca-44ac-b9f2-6d7797184f55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:20:07.394616 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.394593 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-kube-api-access-nj9ld" pod "56bee5e7-e3ca-44ac-b9f2-6d7797184f55" (UID: "56bee5e7-e3ca-44ac-b9f2-6d7797184f55"). InnerVolumeSpecName "kube-api-access-nj9ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:20:07.394616 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.394590 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-xks-ca" pod "56bee5e7-e3ca-44ac-b9f2-6d7797184f55" (UID: "56bee5e7-e3ca-44ac-b9f2-6d7797184f55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:20:07.494289 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.494217 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:07.494289 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.494246 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9ld\" (UniqueName: \"kubernetes.io/projected/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-kube-api-access-nj9ld\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:07.494289 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:07.494257 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bee5e7-e3ca-44ac-b9f2-6d7797184f55-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:08.226481 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:08.226454 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" Oct 31 20:20:08.226654 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:08.226492 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt" event={"ID":"56bee5e7-e3ca-44ac-b9f2-6d7797184f55","Type":"ContainerDied","Data":"b300c8497ad528fcb76a58b1307b1a46adc4e0488ba626df2ed603d930bf949f"} Oct 31 20:20:08.226654 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:08.226541 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b300c8497ad528fcb76a58b1307b1a46adc4e0488ba626df2ed603d930bf949f" Oct 31 20:20:10.092290 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:10.092253 6481 scope.go:122] "RemoveContainer" containerID="0612611ccfffcc5cfb23103619e8907be93eaf2a567fc8faef714a4ae849aa23" Oct 31 20:20:20.746192 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:20.746090 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f9fls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:20:20.746951 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:20.746913 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f9fls" podUID="20b9145e-b5bb-4776-a30f-a904b9e01370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:20:25.043512 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.043476 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt"] Oct 31 20:20:25.094621 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.094584 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ptwvt"] Oct 31 20:20:25.166637 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.166600 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j"] Oct 31 20:20:25.167067 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.167054 6481 cpu_manager.go:414] "containerMap: removing container" podUID="56bee5e7-e3ca-44ac-b9f2-6d7797184f55" containerName="revoke-keys" Oct 31 20:20:25.167121 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.167069 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bee5e7-e3ca-44ac-b9f2-6d7797184f55" containerName="revoke-keys" Oct 31 20:20:25.167162 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.167151 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bee5e7-e3ca-44ac-b9f2-6d7797184f55" containerName="revoke-keys" Oct 31 20:20:25.170204 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.170186 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.223627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.223588 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j"] Oct 31 20:20:25.223964 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.223942 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:20:25.224132 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.224110 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:20:25.320173 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.320061 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmhb\" (UniqueName: \"kubernetes.io/projected/ce974d0e-16ef-454a-bc2e-8bd75af99965-kube-api-access-wvmhb\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.320366 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.320277 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.320436 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.320383 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.421339 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.421295 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmhb\" (UniqueName: \"kubernetes.io/projected/ce974d0e-16ef-454a-bc2e-8bd75af99965-kube-api-access-wvmhb\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.421529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.421446 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.421529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.421509 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.422212 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.422186 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.423826 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.423805 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.480489 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.480454 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmhb\" (UniqueName: \"kubernetes.io/projected/ce974d0e-16ef-454a-bc2e-8bd75af99965-kube-api-access-wvmhb\") pod \"maas-api-revoke-sub-3d4156f1-7t58j\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.483285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.483264 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:25.850717 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:20:25.850683 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce974d0e_16ef_454a_bc2e_8bd75af99965.slice/crio-f414ced1965f173e8d5b836a2bed95ed0b3639faf06fa0f67d4fa3bedf5cb222 WatchSource:0}: Error finding container f414ced1965f173e8d5b836a2bed95ed0b3639faf06fa0f67d4fa3bedf5cb222: Status 404 returned error can't find the container with id f414ced1965f173e8d5b836a2bed95ed0b3639faf06fa0f67d4fa3bedf5cb222 Oct 31 20:20:25.854893 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:25.854870 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j"] Oct 31 20:20:26.339084 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:26.339044 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" event={"ID":"ce974d0e-16ef-454a-bc2e-8bd75af99965","Type":"ContainerStarted","Data":"94848f95df31851c23e77b6d4186eca28462968a25e5a117fe2f498a658cf4f9"} Oct 31 20:20:26.339523 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:26.339104 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" event={"ID":"ce974d0e-16ef-454a-bc2e-8bd75af99965","Type":"ContainerStarted","Data":"f414ced1965f173e8d5b836a2bed95ed0b3639faf06fa0f67d4fa3bedf5cb222"} Oct 31 20:20:26.457534 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:26.457428 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" podStartSLOduration=1.457413746 podStartE2EDuration="1.457413746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:20:26.456343811 +0000 UTC m=+3797.696566940" watchObservedRunningTime="2026-10-31 20:20:26.457413746 +0000 UTC m=+3797.697636875" Oct 31 20:20:26.878816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:26.878778 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bee5e7-e3ca-44ac-b9f2-6d7797184f55" path="/var/lib/kubelet/pods/56bee5e7-e3ca-44ac-b9f2-6d7797184f55/volumes" Oct 31 20:20:27.325647 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:20:27.325610 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:20:27.346439 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:27.346403 6481 generic.go:356] "Generic (PLEG): container finished" podID="ce974d0e-16ef-454a-bc2e-8bd75af99965" containerID="94848f95df31851c23e77b6d4186eca28462968a25e5a117fe2f498a658cf4f9" exitCode=0 Oct 31 20:20:27.346831 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:27.346483 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" event={"ID":"ce974d0e-16ef-454a-bc2e-8bd75af99965","Type":"ContainerDied","Data":"94848f95df31851c23e77b6d4186eca28462968a25e5a117fe2f498a658cf4f9"} Oct 31 20:20:28.490446 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.490421 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:28.654824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.654735 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-xks-ca\") pod \"ce974d0e-16ef-454a-bc2e-8bd75af99965\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " Oct 31 20:20:28.654824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.654789 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-openshift-ca\") pod \"ce974d0e-16ef-454a-bc2e-8bd75af99965\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " Oct 31 20:20:28.655043 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.654846 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce974d0e-16ef-454a-bc2e-8bd75af99965-kube-api-access-wvmhb\" (UniqueName: \"kubernetes.io/projected/ce974d0e-16ef-454a-bc2e-8bd75af99965-kube-api-access-wvmhb\") pod \"ce974d0e-16ef-454a-bc2e-8bd75af99965\" (UID: \"ce974d0e-16ef-454a-bc2e-8bd75af99965\") " Oct 31 20:20:28.655204 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.655179 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-openshift-ca" pod "ce974d0e-16ef-454a-bc2e-8bd75af99965" (UID: "ce974d0e-16ef-454a-bc2e-8bd75af99965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:20:28.655274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.655261 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:28.656981 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.656962 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-xks-ca" pod "ce974d0e-16ef-454a-bc2e-8bd75af99965" (UID: "ce974d0e-16ef-454a-bc2e-8bd75af99965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:20:28.657153 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.657131 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce974d0e-16ef-454a-bc2e-8bd75af99965-kube-api-access-wvmhb" pod "ce974d0e-16ef-454a-bc2e-8bd75af99965" (UID: "ce974d0e-16ef-454a-bc2e-8bd75af99965"). InnerVolumeSpecName "kube-api-access-wvmhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:20:28.756193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.756153 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmhb\" (UniqueName: \"kubernetes.io/projected/ce974d0e-16ef-454a-bc2e-8bd75af99965-kube-api-access-wvmhb\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:28.756193 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:28.756187 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce974d0e-16ef-454a-bc2e-8bd75af99965-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:29.357116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:29.357082 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" Oct 31 20:20:29.357116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:29.357105 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j" event={"ID":"ce974d0e-16ef-454a-bc2e-8bd75af99965","Type":"ContainerDied","Data":"f414ced1965f173e8d5b836a2bed95ed0b3639faf06fa0f67d4fa3bedf5cb222"} Oct 31 20:20:29.357369 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:29.357154 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f414ced1965f173e8d5b836a2bed95ed0b3639faf06fa0f67d4fa3bedf5cb222" Oct 31 20:20:57.325005 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:20:57.324959 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:20:59.400881 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.400805 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5"] Oct 31 20:20:59.401291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.401249 6481 cpu_manager.go:414] "containerMap: removing container" podUID="ce974d0e-16ef-454a-bc2e-8bd75af99965" containerName="revoke-keys" Oct 31 20:20:59.401291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.401262 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce974d0e-16ef-454a-bc2e-8bd75af99965" containerName="revoke-keys" Oct 31 20:20:59.401373 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.401333 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce974d0e-16ef-454a-bc2e-8bd75af99965" containerName="revoke-keys" Oct 31 20:20:59.404523 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.404504 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.456601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.456571 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5"] Oct 31 20:20:59.505191 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.505160 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:20:59.505418 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.505402 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:20:59.560590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.560562 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.560754 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.560619 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.560803 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.560763 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b7kc\" (UniqueName: \"kubernetes.io/projected/02c0a218-4e58-468e-b3ec-20a51406d05b-kube-api-access-7b7kc\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.661392 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.661292 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7kc\" (UniqueName: \"kubernetes.io/projected/02c0a218-4e58-468e-b3ec-20a51406d05b-kube-api-access-7b7kc\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.661392 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.661383 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.661614 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.661448 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.662072 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.662049 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.663789 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.663762 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:20:59.719554 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:20:59.719522 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7kc\" (UniqueName: \"kubernetes.io/projected/02c0a218-4e58-468e-b3ec-20a51406d05b-kube-api-access-7b7kc\") pod \"maas-api-revoke-sub-e2fc9c5b-96rh5\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:21:00.017301 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:00.017267 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:21:00.381920 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:21:00.381887 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c0a218_4e58_468e_b3ec_20a51406d05b.slice/crio-d3e0a8dbe80b1b055b96dc87192d7e61f78ce201b6da9211fa97e85993a11528 WatchSource:0}: Error finding container d3e0a8dbe80b1b055b96dc87192d7e61f78ce201b6da9211fa97e85993a11528: Status 404 returned error can't find the container with id d3e0a8dbe80b1b055b96dc87192d7e61f78ce201b6da9211fa97e85993a11528 Oct 31 20:21:00.384528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:00.384507 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5"] Oct 31 20:21:00.536268 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:00.536235 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" event={"ID":"02c0a218-4e58-468e-b3ec-20a51406d05b","Type":"ContainerStarted","Data":"d3e0a8dbe80b1b055b96dc87192d7e61f78ce201b6da9211fa97e85993a11528"} Oct 31 20:21:01.544500 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:01.544407 6481 generic.go:356] "Generic (PLEG): container finished" podID="02c0a218-4e58-468e-b3ec-20a51406d05b" containerID="93cf2d813562273f44f9c35f38c3c76906473c60e8570f736fb8d2847354dd2b" exitCode=0 Oct 31 20:21:01.544500 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:01.544448 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" event={"ID":"02c0a218-4e58-468e-b3ec-20a51406d05b","Type":"ContainerDied","Data":"93cf2d813562273f44f9c35f38c3c76906473c60e8570f736fb8d2847354dd2b"} Oct 31 20:21:02.684020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.683973 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:21:02.792206 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.792174 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-openshift-ca\") pod \"02c0a218-4e58-468e-b3ec-20a51406d05b\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " Oct 31 20:21:02.792403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.792235 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c0a218-4e58-468e-b3ec-20a51406d05b-kube-api-access-7b7kc\" (UniqueName: \"kubernetes.io/projected/02c0a218-4e58-468e-b3ec-20a51406d05b-kube-api-access-7b7kc\") pod \"02c0a218-4e58-468e-b3ec-20a51406d05b\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " Oct 31 20:21:02.792403 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.792298 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-xks-ca\") pod \"02c0a218-4e58-468e-b3ec-20a51406d05b\" (UID: \"02c0a218-4e58-468e-b3ec-20a51406d05b\") " Oct 31 20:21:02.792588 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.792558 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-openshift-ca" pod "02c0a218-4e58-468e-b3ec-20a51406d05b" (UID: "02c0a218-4e58-468e-b3ec-20a51406d05b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:21:02.794223 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.794187 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c0a218-4e58-468e-b3ec-20a51406d05b-kube-api-access-7b7kc" pod "02c0a218-4e58-468e-b3ec-20a51406d05b" (UID: "02c0a218-4e58-468e-b3ec-20a51406d05b"). InnerVolumeSpecName "kube-api-access-7b7kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:21:02.794422 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.794400 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-xks-ca" pod "02c0a218-4e58-468e-b3ec-20a51406d05b" (UID: "02c0a218-4e58-468e-b3ec-20a51406d05b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:21:02.893940 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.893910 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:02.893940 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.893939 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b7kc\" (UniqueName: \"kubernetes.io/projected/02c0a218-4e58-468e-b3ec-20a51406d05b-kube-api-access-7b7kc\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:02.894138 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:02.893955 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c0a218-4e58-468e-b3ec-20a51406d05b-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:03.555949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:03.555918 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" Oct 31 20:21:03.555949 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:03.555938 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5" event={"ID":"02c0a218-4e58-468e-b3ec-20a51406d05b","Type":"ContainerDied","Data":"d3e0a8dbe80b1b055b96dc87192d7e61f78ce201b6da9211fa97e85993a11528"} Oct 31 20:21:03.556186 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:03.555982 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3e0a8dbe80b1b055b96dc87192d7e61f78ce201b6da9211fa97e85993a11528" Oct 31 20:21:10.112350 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.112313 6481 scope.go:122] "RemoveContainer" containerID="37edaef36e3a8115076edaa832dd96608cc885895f97df4ed551d7ce3cfa042d" Oct 31 20:21:10.191239 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.191211 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h"] Oct 31 20:21:10.191611 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.191599 6481 cpu_manager.go:414] "containerMap: removing container" podUID="02c0a218-4e58-468e-b3ec-20a51406d05b" containerName="revoke-keys" Oct 31 20:21:10.191661 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.191613 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c0a218-4e58-468e-b3ec-20a51406d05b" containerName="revoke-keys" Oct 31 20:21:10.191712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.191701 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c0a218-4e58-468e-b3ec-20a51406d05b" containerName="revoke-keys" Oct 31 20:21:10.196064 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.196045 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.247777 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.247749 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h"] Oct 31 20:21:10.256575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.256545 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.256707 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.256590 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8h2l\" (UniqueName: \"kubernetes.io/projected/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-kube-api-access-z8h2l\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.256707 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.256668 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.296557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.296530 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:21:10.296680 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.296621 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:21:10.358196 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.358157 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.358391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.358215 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h2l\" (UniqueName: \"kubernetes.io/projected/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-kube-api-access-z8h2l\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.358391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.358244 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.358832 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.358806 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.360621 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.360599 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.416554 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.416497 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8h2l\" (UniqueName: \"kubernetes.io/projected/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-kube-api-access-z8h2l\") pod \"maas-api-revoke-sub-5a29efb3-qhk9h\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.508910 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.508878 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:10.875502 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:21:10.875469 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c11bccc_5bdb_4084_85d0_b707fbd2ea6b.slice/crio-718900c5f7821d0dea0fd2a79d8d8d08de2025346165b7a8f0cf9007bfd3718d WatchSource:0}: Error finding container 718900c5f7821d0dea0fd2a79d8d8d08de2025346165b7a8f0cf9007bfd3718d: Status 404 returned error can't find the container with id 718900c5f7821d0dea0fd2a79d8d8d08de2025346165b7a8f0cf9007bfd3718d Oct 31 20:21:10.880221 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:10.880198 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h"] Oct 31 20:21:11.600617 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:11.600584 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" event={"ID":"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b","Type":"ContainerStarted","Data":"3dad82935160243df390dc3c1278fd287d1021522e8e80b5acce0a7a75dc31f3"} Oct 31 20:21:11.601030 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:11.600637 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" event={"ID":"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b","Type":"ContainerStarted","Data":"718900c5f7821d0dea0fd2a79d8d8d08de2025346165b7a8f0cf9007bfd3718d"} Oct 31 20:21:11.718786 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:11.718738 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" podStartSLOduration=1.718723563 podStartE2EDuration="1.718723563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:21:11.716432795 +0000 UTC m=+3842.956655924" watchObservedRunningTime="2026-10-31 20:21:11.718723563 +0000 UTC m=+3842.958946700" Oct 31 20:21:12.611099 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:12.611058 6481 generic.go:356] "Generic (PLEG): container finished" podID="3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" containerID="3dad82935160243df390dc3c1278fd287d1021522e8e80b5acce0a7a75dc31f3" exitCode=0 Oct 31 20:21:12.611739 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:12.611191 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" event={"ID":"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b","Type":"ContainerDied","Data":"3dad82935160243df390dc3c1278fd287d1021522e8e80b5acce0a7a75dc31f3"} Oct 31 20:21:14.324212 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.324177 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:14.463063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.438340 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-openshift-ca\") pod \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " Oct 31 20:21:14.463063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.438450 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-xks-ca\") pod \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " Oct 31 20:21:14.463063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.438568 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-kube-api-access-z8h2l\" (UniqueName: \"kubernetes.io/projected/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-kube-api-access-z8h2l\") pod \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\" (UID: \"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b\") " Oct 31 20:21:14.463063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.439738 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-openshift-ca" pod "3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" (UID: "3c11bccc-5bdb-4084-85d0-b707fbd2ea6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:21:14.557621 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.557580 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:14.617019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.616206 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-kube-api-access-z8h2l" pod "3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" (UID: "3c11bccc-5bdb-4084-85d0-b707fbd2ea6b"). InnerVolumeSpecName "kube-api-access-z8h2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:21:14.636766 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.636703 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-xks-ca" pod "3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" (UID: "3c11bccc-5bdb-4084-85d0-b707fbd2ea6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:21:14.696249 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.696178 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:14.696249 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.696213 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8h2l\" (UniqueName: \"kubernetes.io/projected/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b-kube-api-access-z8h2l\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:14.713200 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.713097 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" event={"ID":"3c11bccc-5bdb-4084-85d0-b707fbd2ea6b","Type":"ContainerDied","Data":"718900c5f7821d0dea0fd2a79d8d8d08de2025346165b7a8f0cf9007bfd3718d"} Oct 31 20:21:14.713454 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.713432 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718900c5f7821d0dea0fd2a79d8d8d08de2025346165b7a8f0cf9007bfd3718d" Oct 31 20:21:14.713647 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:14.713631 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h" Oct 31 20:21:27.047156 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:27.047117 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr"] Oct 31 20:21:27.098010 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:27.097959 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7gmmr"] Oct 31 20:21:27.323674 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:21:27.323591 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:21:28.731284 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.731249 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j"] Oct 31 20:21:28.782233 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.782194 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7t58j"] Oct 31 20:21:28.855658 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.855621 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v"] Oct 31 20:21:28.856050 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.856036 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" containerName="revoke-keys" Oct 31 20:21:28.856105 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.856051 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" containerName="revoke-keys" Oct 31 20:21:28.856141 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.856129 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" containerName="revoke-keys" Oct 31 20:21:28.859476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.859459 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:28.877624 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.877602 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a934192d-c121-4aad-874d-71aa80a6a875" path="/var/lib/kubelet/pods/a934192d-c121-4aad-874d-71aa80a6a875/volumes" Oct 31 20:21:28.878046 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.878030 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce974d0e-16ef-454a-bc2e-8bd75af99965" path="/var/lib/kubelet/pods/ce974d0e-16ef-454a-bc2e-8bd75af99965/volumes" Oct 31 20:21:28.912762 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.912731 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v"] Oct 31 20:21:28.912952 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.912934 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:21:28.913063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.912983 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:21:28.954298 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.954266 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:28.954475 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.954303 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skpns\" (UniqueName: \"kubernetes.io/projected/6988aba4-8e1e-4d65-b204-1c7889f70a33-kube-api-access-skpns\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:28.954475 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:28.954460 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.055255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.055164 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.055444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.055271 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.055444 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.055302 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpns\" (UniqueName: \"kubernetes.io/projected/6988aba4-8e1e-4d65-b204-1c7889f70a33-kube-api-access-skpns\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.055818 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.055795 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.057599 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.057576 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.113228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.113205 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skpns\" (UniqueName: \"kubernetes.io/projected/6988aba4-8e1e-4d65-b204-1c7889f70a33-kube-api-access-skpns\") pod \"maas-api-revoke-sub-3d4156f1-9s49v\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.172664 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.172641 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:29.537599 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:21:29.537569 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6988aba4_8e1e_4d65_b204_1c7889f70a33.slice/crio-6f05b8bac697314d07446cf3814a169952ef86135ee331c0afbe2ebd4c14c783 WatchSource:0}: Error finding container 6f05b8bac697314d07446cf3814a169952ef86135ee331c0afbe2ebd4c14c783: Status 404 returned error can't find the container with id 6f05b8bac697314d07446cf3814a169952ef86135ee331c0afbe2ebd4c14c783 Oct 31 20:21:29.540044 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.539935 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v"] Oct 31 20:21:29.799777 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:29.799693 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" event={"ID":"6988aba4-8e1e-4d65-b204-1c7889f70a33","Type":"ContainerStarted","Data":"6f05b8bac697314d07446cf3814a169952ef86135ee331c0afbe2ebd4c14c783"} Oct 31 20:21:30.807502 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:30.807465 6481 generic.go:356] "Generic (PLEG): container finished" podID="6988aba4-8e1e-4d65-b204-1c7889f70a33" containerID="70d6d85f909134bc0c79f7a2b23fdc5c166af9358e84c3cbe7d6a5f8e2f03364" exitCode=0 Oct 31 20:21:30.807900 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:30.807512 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" event={"ID":"6988aba4-8e1e-4d65-b204-1c7889f70a33","Type":"ContainerDied","Data":"70d6d85f909134bc0c79f7a2b23fdc5c166af9358e84c3cbe7d6a5f8e2f03364"} Oct 31 20:21:31.951314 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:31.951292 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:31.976587 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:31.976565 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-openshift-ca\") pod \"6988aba4-8e1e-4d65-b204-1c7889f70a33\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " Oct 31 20:21:31.976707 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:31.976694 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-xks-ca\") pod \"6988aba4-8e1e-4d65-b204-1c7889f70a33\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " Oct 31 20:21:31.976754 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:31.976734 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6988aba4-8e1e-4d65-b204-1c7889f70a33-kube-api-access-skpns\" (UniqueName: \"kubernetes.io/projected/6988aba4-8e1e-4d65-b204-1c7889f70a33-kube-api-access-skpns\") pod \"6988aba4-8e1e-4d65-b204-1c7889f70a33\" (UID: \"6988aba4-8e1e-4d65-b204-1c7889f70a33\") " Oct 31 20:21:31.976903 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:31.976880 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-openshift-ca" pod "6988aba4-8e1e-4d65-b204-1c7889f70a33" (UID: "6988aba4-8e1e-4d65-b204-1c7889f70a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:21:31.978795 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:31.978769 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-xks-ca" pod "6988aba4-8e1e-4d65-b204-1c7889f70a33" (UID: "6988aba4-8e1e-4d65-b204-1c7889f70a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:21:31.978901 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:31.978839 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6988aba4-8e1e-4d65-b204-1c7889f70a33-kube-api-access-skpns" pod "6988aba4-8e1e-4d65-b204-1c7889f70a33" (UID: "6988aba4-8e1e-4d65-b204-1c7889f70a33"). InnerVolumeSpecName "kube-api-access-skpns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:21:32.078448 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:32.078373 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:32.078448 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:32.078412 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skpns\" (UniqueName: \"kubernetes.io/projected/6988aba4-8e1e-4d65-b204-1c7889f70a33-kube-api-access-skpns\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:32.078448 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:32.078430 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6988aba4-8e1e-4d65-b204-1c7889f70a33-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:21:32.818281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:32.818251 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" Oct 31 20:21:32.818472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:32.818285 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v" event={"ID":"6988aba4-8e1e-4d65-b204-1c7889f70a33","Type":"ContainerDied","Data":"6f05b8bac697314d07446cf3814a169952ef86135ee331c0afbe2ebd4c14c783"} Oct 31 20:21:32.818472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:32.818336 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f05b8bac697314d07446cf3814a169952ef86135ee331c0afbe2ebd4c14c783" Oct 31 20:21:45.306989 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.306953 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c"] Oct 31 20:21:45.307441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.307389 6481 cpu_manager.go:414] "containerMap: removing container" podUID="6988aba4-8e1e-4d65-b204-1c7889f70a33" containerName="revoke-keys" Oct 31 20:21:45.307441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.307400 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6988aba4-8e1e-4d65-b204-1c7889f70a33" containerName="revoke-keys" Oct 31 20:21:45.307548 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.307482 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6988aba4-8e1e-4d65-b204-1c7889f70a33" containerName="revoke-keys" Oct 31 20:21:45.310590 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.310574 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.364287 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.364263 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b50005-openshift-default-dockercfg-kfcfz\"" Oct 31 20:21:45.405173 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405132 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-socket\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405200 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708fd31c-ed11-4ed2-a252-803fe1066227-istio-podinfo\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405356 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-credential-socket\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405441 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405402 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-envoy\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405501 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-data\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405541 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mb6r\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-kube-api-access-7mb6r\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405577 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708fd31c-ed11-4ed2-a252-803fe1066227-istiod-ca-cert\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405607 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-certs\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.405844 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.405657 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-istio-token\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.417513 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.417485 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c"] Oct 31 20:21:45.506358 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506320 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-istio-token\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506466 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-socket\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506494 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708fd31c-ed11-4ed2-a252-803fe1066227-istio-podinfo\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506515 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-credential-socket\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506532 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-envoy\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506895 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506569 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-data\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506895 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506600 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mb6r\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-kube-api-access-7mb6r\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506895 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506629 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708fd31c-ed11-4ed2-a252-803fe1066227-istiod-ca-cert\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.506895 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506664 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-certs\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.507163 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506952 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-socket\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.507163 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.506981 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-credential-socket\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.507262 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.507202 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-data\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.507318 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.507272 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-certs\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.507478 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.507457 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708fd31c-ed11-4ed2-a252-803fe1066227-istiod-ca-cert\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.509065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.509042 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-envoy\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.509339 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.509317 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708fd31c-ed11-4ed2-a252-803fe1066227-istio-podinfo\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.564912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.564837 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mb6r\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-kube-api-access-7mb6r\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.565084 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.564984 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-istio-token\") pod \"e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:45.623443 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:45.623407 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:46.036359 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:21:46.036330 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708fd31c_ed11_4ed2_a252_803fe1066227.slice/crio-330dfdc33c178fca55d578195098a3e1ff9aa2e38e0bec0d922d717116add2f4 WatchSource:0}: Error finding container 330dfdc33c178fca55d578195098a3e1ff9aa2e38e0bec0d922d717116add2f4: Status 404 returned error can't find the container with id 330dfdc33c178fca55d578195098a3e1ff9aa2e38e0bec0d922d717116add2f4 Oct 31 20:21:46.038571 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:46.038538 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:21:46.038665 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:46.038606 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:21:46.038665 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:46.038634 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:21:46.039830 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:46.039805 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c"] Oct 31 20:21:46.890735 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:46.890693 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" event={"ID":"708fd31c-ed11-4ed2-a252-803fe1066227","Type":"ContainerStarted","Data":"8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed"} Oct 31 20:21:46.891177 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:46.890756 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" event={"ID":"708fd31c-ed11-4ed2-a252-803fe1066227","Type":"ContainerStarted","Data":"330dfdc33c178fca55d578195098a3e1ff9aa2e38e0bec0d922d717116add2f4"} Oct 31 20:21:47.062469 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:47.062415 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" podStartSLOduration=2.062400288 podStartE2EDuration="2.062400288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:21:47.060690898 +0000 UTC m=+3878.300914027" watchObservedRunningTime="2026-10-31 20:21:47.062400288 +0000 UTC m=+3878.302623416" Oct 31 20:21:47.623605 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:47.623568 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:47.628797 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:47.628770 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:47.897891 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:47.897806 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:47.898698 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:47.898685 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:21:56.349174 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.349139 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm"] Oct 31 20:21:56.353028 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.353005 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.407023 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.406964 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b50005-serving-cert\"" Oct 31 20:21:56.417610 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.417572 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf00c874-e823-4404-996f-5f2605371ff8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.417739 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.417640 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6t8\" (UniqueName: \"kubernetes.io/projected/cf00c874-e823-4404-996f-5f2605371ff8-kube-api-access-wl6t8\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.417739 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.417721 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.417877 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.417853 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.458086 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.458054 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 20:21:56.458263 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.458134 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xd7wl\"" Oct 31 20:21:56.460493 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.460470 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm"] Oct 31 20:21:56.519175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.519138 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.519362 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.519196 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf00c874-e823-4404-996f-5f2605371ff8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.519362 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.519231 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6t8\" (UniqueName: \"kubernetes.io/projected/cf00c874-e823-4404-996f-5f2605371ff8-kube-api-access-wl6t8\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.519362 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.519264 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.519362 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:21:56.519312 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-serving-cert: secret "maas-api-e2e-worker-single-main-b50005-serving-cert" not found Oct 31 20:21:56.519512 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:21:56.519406 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls podName:cf00c874-e823-4404-996f-5f2605371ff8 nodeName:}" failed. No retries permitted until 2026-10-31 20:21:57.019384555 +0000 UTC m=+3888.259607676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls") pod "maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" (UID: "cf00c874-e823-4404-996f-5f2605371ff8") : secret "maas-api-e2e-worker-single-main-b50005-serving-cert" not found Oct 31 20:21:56.519941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.519913 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf00c874-e823-4404-996f-5f2605371ff8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.521869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.521844 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:56.578601 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:56.578565 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6t8\" (UniqueName: \"kubernetes.io/projected/cf00c874-e823-4404-996f-5f2605371ff8-kube-api-access-wl6t8\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:57.024667 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:57.024633 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:57.027061 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:57.027038 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:57.266786 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:57.266751 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:57.328580 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:21:57.328546 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:21:57.638556 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:21:57.638526 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf00c874_e823_4404_996f_5f2605371ff8.slice/crio-bd727d60fe01a9ee4f76ef0f46fef7e9e0bc3afc8091a4a858b1ce698ec32540 WatchSource:0}: Error finding container bd727d60fe01a9ee4f76ef0f46fef7e9e0bc3afc8091a4a858b1ce698ec32540: Status 404 returned error can't find the container with id bd727d60fe01a9ee4f76ef0f46fef7e9e0bc3afc8091a4a858b1ce698ec32540 Oct 31 20:21:57.640175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:57.640154 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:21:57.640985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:57.640962 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm"] Oct 31 20:21:57.952428 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:57.952349 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" event={"ID":"cf00c874-e823-4404-996f-5f2605371ff8","Type":"ContainerStarted","Data":"bd727d60fe01a9ee4f76ef0f46fef7e9e0bc3afc8091a4a858b1ce698ec32540"} Oct 31 20:21:58.047431 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:58.047392 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766"] Oct 31 20:21:58.098440 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:58.098411 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b7766"] Oct 31 20:21:58.878650 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:58.878572 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d629e1ff-ed7f-490a-ad03-947fa8f89dab" path="/var/lib/kubelet/pods/d629e1ff-ed7f-490a-ad03-947fa8f89dab/volumes" Oct 31 20:21:58.959184 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:58.959153 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" event={"ID":"cf00c874-e823-4404-996f-5f2605371ff8","Type":"ContainerStarted","Data":"beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0"} Oct 31 20:21:58.959350 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:58.959260 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:21:59.076047 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:21:59.075988 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" podStartSLOduration=2.090670862 podStartE2EDuration="3.075974958s" totalImagesPullingTime="985.304096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:21:59.075216865 +0000 UTC m=+3890.315439994" watchObservedRunningTime="2026-10-31 20:21:59.075974958 +0000 UTC m=+3890.316198138" Oct 31 20:22:04.993908 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:04.993866 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:22:10.048603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:10.048570 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n"] Oct 31 20:22:10.099622 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:10.099592 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-kjz5n"] Oct 31 20:22:10.879586 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:10.879550 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68689370-f241-4573-9b44-781c098c1720" path="/var/lib/kubelet/pods/68689370-f241-4573-9b44-781c098c1720/volumes" Oct 31 20:22:14.044801 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:14.044765 6481 generic.go:356] "Generic (PLEG): container finished" podID="708fd31c-ed11-4ed2-a252-803fe1066227" containerID="8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed" exitCode=0 Oct 31 20:22:14.045263 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:14.044847 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" event={"ID":"708fd31c-ed11-4ed2-a252-803fe1066227","Type":"ContainerDied","Data":"8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed"} Oct 31 20:22:14.045343 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:14.045326 6481 scope.go:122] "RemoveContainer" containerID="8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed" Oct 31 20:22:14.046586 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:14.046559 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:22:14.046720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:14.046640 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:22:14.046720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:14.046700 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:22:15.051832 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:15.051793 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" event={"ID":"708fd31c-ed11-4ed2-a252-803fe1066227","Type":"ContainerStarted","Data":"35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa"} Oct 31 20:22:15.623948 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:15.623908 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:22:15.625083 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:15.625055 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 20:22:15.625221 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:15.625104 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 20:22:16.624621 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:16.624583 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 20:22:16.625043 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:16.624649 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 20:22:17.624268 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:17.624233 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 20:22:17.624454 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:17.624300 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 20:22:17.898336 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:17.898239 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:22:18.624529 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:18.624497 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 20:22:18.624707 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:18.624544 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 20:22:19.624084 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:19.624051 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 20:22:19.624486 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:19.624102 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 20:22:20.627725 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:20.627695 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:22:20.628774 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:20.628752 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:22:27.323976 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:22:27.323936 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:22:45.047308 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:45.047270 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv"] Oct 31 20:22:45.098648 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:45.098609 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5c6rv"] Oct 31 20:22:46.878291 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:46.878256 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdd75e0-0416-4d87-b391-e820165036e7" path="/var/lib/kubelet/pods/0bdd75e0-0416-4d87-b391-e820165036e7/volumes" Oct 31 20:22:56.050228 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:56.050194 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg"] Oct 31 20:22:56.101331 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:56.101299 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9p5dg"] Oct 31 20:22:56.879254 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:22:56.879216 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb01ad0-881c-4567-858e-4d9be951ebae" path="/var/lib/kubelet/pods/ddb01ad0-881c-4567-858e-4d9be951ebae/volumes" Oct 31 20:22:57.323779 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:22:57.323742 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:23:10.146558 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:10.146527 6481 scope.go:122] "RemoveContainer" containerID="1f6557e2010e0a833079344e6a0536d2280d419b1b87111e5e22c55a66fbf54d" Oct 31 20:23:10.155254 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:10.155232 6481 scope.go:122] "RemoveContainer" containerID="c464cfab0cdd186c582ac04ae4855e662f250899b673ee6f516a3908ed531020" Oct 31 20:23:10.162780 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:10.162764 6481 scope.go:122] "RemoveContainer" containerID="037e96dd68b49bb89b6aeea69936ce08478ce62e6b70174f7b7f5c30d61ad547" Oct 31 20:23:27.324313 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:23:27.324227 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:23:31.050270 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:31.050236 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n"] Oct 31 20:23:31.101457 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:31.101422 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-s4x5n"] Oct 31 20:23:32.878255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:32.878220 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d138e030-c19b-473e-a17e-86753acb09e3" path="/var/lib/kubelet/pods/d138e030-c19b-473e-a17e-86753acb09e3/volumes" Oct 31 20:23:34.452456 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:34.452423 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm"] Oct 31 20:23:34.452938 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:34.452667 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" podUID="cf00c874-e823-4404-996f-5f2605371ff8" containerName="maas-api" containerID="cri-o://beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0" gracePeriod=30 Oct 31 20:23:34.964451 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:34.964403 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" podUID="cf00c874-e823-4404-996f-5f2605371ff8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.162:8443/health\": dial tcp 10.134.0.162:8443: connect: connection refused" Oct 31 20:23:35.701176 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.701146 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:23:35.792046 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.791952 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-metrics-certs\") pod \"cf00c874-e823-4404-996f-5f2605371ff8\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " Oct 31 20:23:35.792046 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.792025 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf00c874-e823-4404-996f-5f2605371ff8-kube-api-access-wl6t8\" (UniqueName: \"kubernetes.io/projected/cf00c874-e823-4404-996f-5f2605371ff8-kube-api-access-wl6t8\") pod \"cf00c874-e823-4404-996f-5f2605371ff8\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " Oct 31 20:23:35.792301 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.792080 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls\") pod \"cf00c874-e823-4404-996f-5f2605371ff8\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " Oct 31 20:23:35.792301 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.792103 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf00c874-e823-4404-996f-5f2605371ff8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf00c874-e823-4404-996f-5f2605371ff8-openshift-service-ca\") pod \"cf00c874-e823-4404-996f-5f2605371ff8\" (UID: \"cf00c874-e823-4404-996f-5f2605371ff8\") " Oct 31 20:23:35.792521 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.792490 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf00c874-e823-4404-996f-5f2605371ff8-openshift-service-ca" pod "cf00c874-e823-4404-996f-5f2605371ff8" (UID: "cf00c874-e823-4404-996f-5f2605371ff8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:23:35.794262 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.794229 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-metrics-certs" pod "cf00c874-e823-4404-996f-5f2605371ff8" (UID: "cf00c874-e823-4404-996f-5f2605371ff8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:23:35.794364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.794250 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls" pod "cf00c874-e823-4404-996f-5f2605371ff8" (UID: "cf00c874-e823-4404-996f-5f2605371ff8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:23:35.794364 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.794313 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf00c874-e823-4404-996f-5f2605371ff8-kube-api-access-wl6t8" pod "cf00c874-e823-4404-996f-5f2605371ff8" (UID: "cf00c874-e823-4404-996f-5f2605371ff8"). InnerVolumeSpecName "kube-api-access-wl6t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:35.892942 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.892900 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:35.892942 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.892938 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6t8\" (UniqueName: \"kubernetes.io/projected/cf00c874-e823-4404-996f-5f2605371ff8-kube-api-access-wl6t8\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:35.893202 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.892952 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf00c874-e823-4404-996f-5f2605371ff8-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:35.893202 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:35.892966 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf00c874-e823-4404-996f-5f2605371ff8-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:36.511079 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.511044 6481 generic.go:356] "Generic (PLEG): container finished" podID="cf00c874-e823-4404-996f-5f2605371ff8" containerID="beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0" exitCode=0 Oct 31 20:23:36.511295 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.511088 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" Oct 31 20:23:36.511295 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.511127 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" event={"ID":"cf00c874-e823-4404-996f-5f2605371ff8","Type":"ContainerDied","Data":"beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0"} Oct 31 20:23:36.511295 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.511188 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm" event={"ID":"cf00c874-e823-4404-996f-5f2605371ff8","Type":"ContainerDied","Data":"bd727d60fe01a9ee4f76ef0f46fef7e9e0bc3afc8091a4a858b1ce698ec32540"} Oct 31 20:23:36.511295 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.511216 6481 scope.go:122] "RemoveContainer" containerID="beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0" Oct 31 20:23:36.521274 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.521253 6481 scope.go:122] "RemoveContainer" containerID="beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0" Oct 31 20:23:36.521560 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:23:36.521541 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0\": container with ID starting with beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0 not found: ID does not exist" containerID="beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0" Oct 31 20:23:36.521627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.521567 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0"} err="failed to get container status \"beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0\": rpc error: code = NotFound desc = could not find container \"beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0\": container with ID starting with beda37a48b8153e969a355b61a1257de8508ffb7dafd4762b601cee2e83e7aa0 not found: ID does not exist" Oct 31 20:23:36.677963 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.677929 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm"] Oct 31 20:23:36.729163 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.729127 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b50005-68b4bc5676-hzpkm"] Oct 31 20:23:36.878231 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:36.878151 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf00c874-e823-4404-996f-5f2605371ff8" path="/var/lib/kubelet/pods/cf00c874-e823-4404-996f-5f2605371ff8/volumes" Oct 31 20:23:43.124626 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:43.124578 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c"] Oct 31 20:23:43.125078 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:43.124791 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" containerID="cri-o://35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa" gracePeriod=30 Oct 31 20:23:44.593925 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.593890 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v"] Oct 31 20:23:44.645006 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.644973 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s49v"] Oct 31 20:23:44.718189 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.718156 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j"] Oct 31 20:23:44.718565 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.718553 6481 cpu_manager.go:414] "containerMap: removing container" podUID="cf00c874-e823-4404-996f-5f2605371ff8" containerName="maas-api" Oct 31 20:23:44.718613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.718566 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf00c874-e823-4404-996f-5f2605371ff8" containerName="maas-api" Oct 31 20:23:44.718669 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.718657 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf00c874-e823-4404-996f-5f2605371ff8" containerName="maas-api" Oct 31 20:23:44.721785 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.721764 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.774478 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.774451 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j"] Oct 31 20:23:44.775021 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.774988 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:23:44.775200 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.775162 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:23:44.805856 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.805832 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.805985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.805922 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.806090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.806073 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncnr\" (UniqueName: \"kubernetes.io/projected/38c40422-72db-450f-9a3d-c5faddc9cf91-kube-api-access-8ncnr\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.878712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.878612 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6988aba4-8e1e-4d65-b204-1c7889f70a33" path="/var/lib/kubelet/pods/6988aba4-8e1e-4d65-b204-1c7889f70a33/volumes" Oct 31 20:23:44.907133 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.907096 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.907318 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.907158 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.907318 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.907295 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncnr\" (UniqueName: \"kubernetes.io/projected/38c40422-72db-450f-9a3d-c5faddc9cf91-kube-api-access-8ncnr\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.907870 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.907843 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.909509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.909481 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:44.965331 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:44.965300 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncnr\" (UniqueName: \"kubernetes.io/projected/38c40422-72db-450f-9a3d-c5faddc9cf91-kube-api-access-8ncnr\") pod \"maas-api-revoke-sub-3d4156f1-xwx7j\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:45.034756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:45.034724 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:45.401183 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:23:45.401144 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c40422_72db_450f_9a3d_c5faddc9cf91.slice/crio-18a29d185289acf3a51f5394dbe4f8cdee1e63a29279f129c23d4f39cbc0ecf7 WatchSource:0}: Error finding container 18a29d185289acf3a51f5394dbe4f8cdee1e63a29279f129c23d4f39cbc0ecf7: Status 404 returned error can't find the container with id 18a29d185289acf3a51f5394dbe4f8cdee1e63a29279f129c23d4f39cbc0ecf7 Oct 31 20:23:45.407365 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:45.404127 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j"] Oct 31 20:23:45.556670 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:45.556631 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" event={"ID":"38c40422-72db-450f-9a3d-c5faddc9cf91","Type":"ContainerStarted","Data":"18a29d185289acf3a51f5394dbe4f8cdee1e63a29279f129c23d4f39cbc0ecf7"} Oct 31 20:23:46.564916 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:46.564883 6481 generic.go:356] "Generic (PLEG): container finished" podID="38c40422-72db-450f-9a3d-c5faddc9cf91" containerID="73578ecbc85353a3fa9a4427135baf7c4cc3c6f423deb19b27bf4e3750a839af" exitCode=0 Oct 31 20:23:46.565368 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:46.564966 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" event={"ID":"38c40422-72db-450f-9a3d-c5faddc9cf91","Type":"ContainerDied","Data":"73578ecbc85353a3fa9a4427135baf7c4cc3c6f423deb19b27bf4e3750a839af"} Oct 31 20:23:47.706874 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.706843 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:47.838257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.838183 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-openshift-ca\") pod \"38c40422-72db-450f-9a3d-c5faddc9cf91\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " Oct 31 20:23:47.838257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.838234 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-xks-ca\") pod \"38c40422-72db-450f-9a3d-c5faddc9cf91\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " Oct 31 20:23:47.838466 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.838295 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c40422-72db-450f-9a3d-c5faddc9cf91-kube-api-access-8ncnr\" (UniqueName: \"kubernetes.io/projected/38c40422-72db-450f-9a3d-c5faddc9cf91-kube-api-access-8ncnr\") pod \"38c40422-72db-450f-9a3d-c5faddc9cf91\" (UID: \"38c40422-72db-450f-9a3d-c5faddc9cf91\") " Oct 31 20:23:47.838614 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.838589 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-openshift-ca" pod "38c40422-72db-450f-9a3d-c5faddc9cf91" (UID: "38c40422-72db-450f-9a3d-c5faddc9cf91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:23:47.840447 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.840409 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c40422-72db-450f-9a3d-c5faddc9cf91-kube-api-access-8ncnr" pod "38c40422-72db-450f-9a3d-c5faddc9cf91" (UID: "38c40422-72db-450f-9a3d-c5faddc9cf91"). InnerVolumeSpecName "kube-api-access-8ncnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:47.840603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.840474 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-xks-ca" pod "38c40422-72db-450f-9a3d-c5faddc9cf91" (UID: "38c40422-72db-450f-9a3d-c5faddc9cf91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:23:47.939285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.939252 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ncnr\" (UniqueName: \"kubernetes.io/projected/38c40422-72db-450f-9a3d-c5faddc9cf91-kube-api-access-8ncnr\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:47.939285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.939280 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:47.939285 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:47.939291 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c40422-72db-450f-9a3d-c5faddc9cf91-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.376085 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.376060 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:23:48.444869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.444803 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-envoy\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.444869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.444836 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-kube-api-access-7mb6r\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-kube-api-access-7mb6r\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.444869 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.444862 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/708fd31c-ed11-4ed2-a252-803fe1066227-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708fd31c-ed11-4ed2-a252-803fe1066227-istio-podinfo\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.445143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.444988 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-istio-token\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-istio-token\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.445143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445072 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-socket\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.445143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445111 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-credential-socket\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.445303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445187 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708fd31c-ed11-4ed2-a252-803fe1066227-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708fd31c-ed11-4ed2-a252-803fe1066227-istiod-ca-cert\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.445303 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445266 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-certs\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.445406 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445352 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-data\") pod \"708fd31c-ed11-4ed2-a252-803fe1066227\" (UID: \"708fd31c-ed11-4ed2-a252-803fe1066227\") " Oct 31 20:23:48.445727 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445342 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-socket" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:48.445783 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445637 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708fd31c-ed11-4ed2-a252-803fe1066227-istiod-ca-cert" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:23:48.445920 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445881 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-credential-socket" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:48.445977 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.445955 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-certs" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:48.446136 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.446115 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-data" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:48.447127 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.447090 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/708fd31c-ed11-4ed2-a252-803fe1066227-istio-podinfo" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:23:48.447127 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.447115 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-envoy" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:48.447397 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.447373 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-kube-api-access-7mb6r" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "kube-api-access-7mb6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:48.447397 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.447382 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-istio-token" pod "708fd31c-ed11-4ed2-a252-803fe1066227" (UID: "708fd31c-ed11-4ed2-a252-803fe1066227"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:48.546586 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546551 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546586 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546582 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546586 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546593 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546915 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546607 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mb6r\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-kube-api-access-7mb6r\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546915 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546618 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708fd31c-ed11-4ed2-a252-803fe1066227-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546915 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546628 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708fd31c-ed11-4ed2-a252-803fe1066227-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546915 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546640 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546915 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546648 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708fd31c-ed11-4ed2-a252-803fe1066227-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.546915 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.546666 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708fd31c-ed11-4ed2-a252-803fe1066227-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:48.577033 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.576979 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" Oct 31 20:23:48.577198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.577026 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j" event={"ID":"38c40422-72db-450f-9a3d-c5faddc9cf91","Type":"ContainerDied","Data":"18a29d185289acf3a51f5394dbe4f8cdee1e63a29279f129c23d4f39cbc0ecf7"} Oct 31 20:23:48.577198 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.577086 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18a29d185289acf3a51f5394dbe4f8cdee1e63a29279f129c23d4f39cbc0ecf7" Oct 31 20:23:48.578948 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.578921 6481 generic.go:356] "Generic (PLEG): container finished" podID="708fd31c-ed11-4ed2-a252-803fe1066227" containerID="35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa" exitCode=0 Oct 31 20:23:48.579112 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.578971 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" Oct 31 20:23:48.579112 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.579020 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" event={"ID":"708fd31c-ed11-4ed2-a252-803fe1066227","Type":"ContainerDied","Data":"35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa"} Oct 31 20:23:48.579112 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.579075 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c" event={"ID":"708fd31c-ed11-4ed2-a252-803fe1066227","Type":"ContainerDied","Data":"330dfdc33c178fca55d578195098a3e1ff9aa2e38e0bec0d922d717116add2f4"} Oct 31 20:23:48.579112 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.579104 6481 scope.go:122] "RemoveContainer" containerID="35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa" Oct 31 20:23:48.589107 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.589090 6481 scope.go:122] "RemoveContainer" containerID="8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed" Oct 31 20:23:48.597753 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.597736 6481 scope.go:122] "RemoveContainer" containerID="35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa" Oct 31 20:23:48.598065 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:23:48.598045 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa\": container with ID starting with 35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa not found: ID does not exist" containerID="35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa" Oct 31 20:23:48.598132 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.598074 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa"} err="failed to get container status \"35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa\": rpc error: code = NotFound desc = could not find container \"35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa\": container with ID starting with 35fb5dadc93457104b025017b42631e8dcfe537b81e095d8531ab8830a53d9aa not found: ID does not exist" Oct 31 20:23:48.598132 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.598091 6481 scope.go:122] "RemoveContainer" containerID="8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed" Oct 31 20:23:48.598335 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:23:48.598317 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed\": container with ID starting with 8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed not found: ID does not exist" containerID="8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed" Oct 31 20:23:48.598391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.598346 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed"} err="failed to get container status \"8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed\": rpc error: code = NotFound desc = could not find container \"8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed\": container with ID starting with 8883a2373a88b2cc2a8d10756ce3e83ea41bd1efa52c666006e7be55f886a0ed not found: ID does not exist" Oct 31 20:23:48.764251 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.764227 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c"] Oct 31 20:23:48.765754 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.765732 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b50005-openshift-default-5d4dc44fdfvkg6c"] Oct 31 20:23:48.879825 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:48.879794 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" path="/var/lib/kubelet/pods/708fd31c-ed11-4ed2-a252-803fe1066227/volumes" Oct 31 20:23:57.331313 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:23:57.331270 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:23:59.050615 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:59.050580 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd"] Oct 31 20:23:59.101720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:23:59.101687 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-26mxd"] Oct 31 20:24:00.877971 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:00.877938 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f03823-eaca-4724-9b99-19855a1079a8" path="/var/lib/kubelet/pods/c7f03823-eaca-4724-9b99-19855a1079a8/volumes" Oct 31 20:24:10.190977 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:10.190945 6481 scope.go:122] "RemoveContainer" containerID="b4d6effe025e7ac8e6b068211c1fcb3e3e0782003f36402076d27350c99a296c" Oct 31 20:24:10.199724 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:10.199704 6481 scope.go:122] "RemoveContainer" containerID="9f5f2c8263ea9aa854190dd2b2902dc1ad275c3f40c16bf49cd27317fc08b27d" Oct 31 20:24:27.324460 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:24:27.324415 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:24:30.051726 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:30.051688 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv"] Oct 31 20:24:30.102986 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:30.102948 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v56lv"] Oct 31 20:24:30.878793 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:30.878757 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbf319b-bab3-4716-8dbe-0006c5f32af8" path="/var/lib/kubelet/pods/8dbf319b-bab3-4716-8dbe-0006c5f32af8/volumes" Oct 31 20:24:57.049322 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:57.049245 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm"] Oct 31 20:24:57.100309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:57.100278 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6xdkm"] Oct 31 20:24:57.324223 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:24:57.324136 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:24:58.878746 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:24:58.878707 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54409742-cf47-436b-ad02-6827776112a6" path="/var/lib/kubelet/pods/54409742-cf47-436b-ad02-6827776112a6/volumes" Oct 31 20:25:10.231106 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:10.231075 6481 scope.go:122] "RemoveContainer" containerID="0abe854beef16f5b01155e0f08a3add76695fcb4955329a4c2c33d919ad8dd61" Oct 31 20:25:10.239772 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:10.239751 6481 scope.go:122] "RemoveContainer" containerID="cdc64010a76afbb0c8b94877a40ad9d8c5d73d0ec5ec9062de2d06f83fe41ddf" Oct 31 20:25:27.324743 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:25:27.324706 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:25:31.841619 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.841582 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv"] Oct 31 20:25:31.842039 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.842013 6481 cpu_manager.go:414] "containerMap: removing container" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" Oct 31 20:25:31.842039 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.842028 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" Oct 31 20:25:31.842121 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.842059 6481 cpu_manager.go:414] "containerMap: removing container" podUID="38c40422-72db-450f-9a3d-c5faddc9cf91" containerName="revoke-keys" Oct 31 20:25:31.842121 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.842066 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c40422-72db-450f-9a3d-c5faddc9cf91" containerName="revoke-keys" Oct 31 20:25:31.842190 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.842153 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" Oct 31 20:25:31.842190 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.842163 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" Oct 31 20:25:31.842190 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.842171 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c40422-72db-450f-9a3d-c5faddc9cf91" containerName="revoke-keys" Oct 31 20:25:31.845212 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.845194 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:31.898952 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.898923 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:25:31.900957 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.900928 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv"] Oct 31 20:25:31.950185 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:31.950158 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:25:32.003847 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.003809 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.004091 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.003859 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.004091 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.004057 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfxp\" (UniqueName: \"kubernetes.io/projected/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-kube-api-access-szfxp\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.104955 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.104865 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.104955 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.104915 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.105254 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.104972 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfxp\" (UniqueName: \"kubernetes.io/projected/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-kube-api-access-szfxp\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.105642 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.105616 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.107319 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.107299 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.163300 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.163273 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfxp\" (UniqueName: \"kubernetes.io/projected/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-kube-api-access-szfxp\") pod \"maas-api-revoke-sub-d576e68b-qpxvv\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.458293 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.458213 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:32.824356 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:25:32.824322 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7da11ba_8ba2_44e0_b2b5_567aadae0fd1.slice/crio-01976e9076043ccb3c7df08969cad99d702fbcede5494313c1e3525a39ca7047 WatchSource:0}: Error finding container 01976e9076043ccb3c7df08969cad99d702fbcede5494313c1e3525a39ca7047: Status 404 returned error can't find the container with id 01976e9076043ccb3c7df08969cad99d702fbcede5494313c1e3525a39ca7047 Oct 31 20:25:32.826546 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:32.826522 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv"] Oct 31 20:25:33.132883 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:33.132843 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" event={"ID":"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1","Type":"ContainerStarted","Data":"adb1307891c31ec9482b3050c973031ba57977e61f8fbab04f591f21e3da402a"} Oct 31 20:25:33.133297 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:33.132898 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" event={"ID":"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1","Type":"ContainerStarted","Data":"01976e9076043ccb3c7df08969cad99d702fbcede5494313c1e3525a39ca7047"} Oct 31 20:25:33.248580 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:33.248530 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" podStartSLOduration=2.248517128 podStartE2EDuration="2.248517128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:33.247394809 +0000 UTC m=+4104.487617938" watchObservedRunningTime="2026-10-31 20:25:33.248517128 +0000 UTC m=+4104.488740271" Oct 31 20:25:34.139819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:34.139781 6481 generic.go:356] "Generic (PLEG): container finished" podID="f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" containerID="adb1307891c31ec9482b3050c973031ba57977e61f8fbab04f591f21e3da402a" exitCode=0 Oct 31 20:25:34.140310 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:34.139870 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" event={"ID":"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1","Type":"ContainerDied","Data":"adb1307891c31ec9482b3050c973031ba57977e61f8fbab04f591f21e3da402a"} Oct 31 20:25:35.278719 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.278695 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:35.438477 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.438406 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-openshift-ca\") pod \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " Oct 31 20:25:35.438477 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.438467 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-xks-ca\") pod \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " Oct 31 20:25:35.438738 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.438495 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-kube-api-access-szfxp\" (UniqueName: \"kubernetes.io/projected/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-kube-api-access-szfxp\") pod \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\" (UID: \"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1\") " Oct 31 20:25:35.438824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.438803 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-openshift-ca" pod "f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" (UID: "f7da11ba-8ba2-44e0-b2b5-567aadae0fd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:25:35.440672 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.440648 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-xks-ca" pod "f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" (UID: "f7da11ba-8ba2-44e0-b2b5-567aadae0fd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:25:35.440768 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.440668 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-kube-api-access-szfxp" pod "f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" (UID: "f7da11ba-8ba2-44e0-b2b5-567aadae0fd1"). InnerVolumeSpecName "kube-api-access-szfxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:25:35.540085 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.540054 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:35.540085 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.540080 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:35.540085 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:35.540089 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szfxp\" (UniqueName: \"kubernetes.io/projected/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1-kube-api-access-szfxp\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:36.151200 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:36.151170 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" Oct 31 20:25:36.151397 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:36.151209 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv" event={"ID":"f7da11ba-8ba2-44e0-b2b5-567aadae0fd1","Type":"ContainerDied","Data":"01976e9076043ccb3c7df08969cad99d702fbcede5494313c1e3525a39ca7047"} Oct 31 20:25:36.151397 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:36.151259 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01976e9076043ccb3c7df08969cad99d702fbcede5494313c1e3525a39ca7047" Oct 31 20:25:52.000230 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:52.000178 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:25:52.000770 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:25:52.000274 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:25:57.324359 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:25:57.324324 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:26:00.637040 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.637006 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m"] Oct 31 20:26:00.637462 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.637404 6481 cpu_manager.go:414] "containerMap: removing container" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" Oct 31 20:26:00.637462 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.637417 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708fd31c-ed11-4ed2-a252-803fe1066227" containerName="istio-proxy" Oct 31 20:26:00.637462 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.637443 6481 cpu_manager.go:414] "containerMap: removing container" podUID="f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" containerName="revoke-keys" Oct 31 20:26:00.637462 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.637448 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" containerName="revoke-keys" Oct 31 20:26:00.637602 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.637515 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" containerName="revoke-keys" Oct 31 20:26:00.640626 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.640610 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.695187 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.694804 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:26:00.696712 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.696683 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m"] Oct 31 20:26:00.744495 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.744467 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:26:00.748940 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.748916 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvf5j\" (UniqueName: \"kubernetes.io/projected/3e5de424-34bd-43d9-8fd4-4bb5acd63224-kube-api-access-pvf5j\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.749090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.748977 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.749158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.749139 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.850248 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.850219 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.850429 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.850305 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvf5j\" (UniqueName: \"kubernetes.io/projected/3e5de424-34bd-43d9-8fd4-4bb5acd63224-kube-api-access-pvf5j\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.850429 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.850342 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.850967 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.850944 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.852592 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.852569 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.911330 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.911263 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvf5j\" (UniqueName: \"kubernetes.io/projected/3e5de424-34bd-43d9-8fd4-4bb5acd63224-kube-api-access-pvf5j\") pod \"maas-api-revoke-sub-d2bc93bc-tm89m\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:00.953855 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:00.953821 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:01.317623 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:26:01.317595 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5de424_34bd_43d9_8fd4_4bb5acd63224.slice/crio-644637d391c2cfba8ed7ca617ae6d49ac80b387d260fe588df2103117050e3ab WatchSource:0}: Error finding container 644637d391c2cfba8ed7ca617ae6d49ac80b387d260fe588df2103117050e3ab: Status 404 returned error can't find the container with id 644637d391c2cfba8ed7ca617ae6d49ac80b387d260fe588df2103117050e3ab Oct 31 20:26:01.320213 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:01.320186 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m"] Oct 31 20:26:02.322568 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:02.322538 6481 generic.go:356] "Generic (PLEG): container finished" podID="3e5de424-34bd-43d9-8fd4-4bb5acd63224" containerID="ac11aa7723e3c0d52bd806fbd7a2ddce45c3eb6dcf54e85e2cffe081dd08094f" exitCode=0 Oct 31 20:26:02.322929 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:02.322592 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" event={"ID":"3e5de424-34bd-43d9-8fd4-4bb5acd63224","Type":"ContainerDied","Data":"ac11aa7723e3c0d52bd806fbd7a2ddce45c3eb6dcf54e85e2cffe081dd08094f"} Oct 31 20:26:02.322929 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:02.322628 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" event={"ID":"3e5de424-34bd-43d9-8fd4-4bb5acd63224","Type":"ContainerStarted","Data":"644637d391c2cfba8ed7ca617ae6d49ac80b387d260fe588df2103117050e3ab"} Oct 31 20:26:03.052308 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.052276 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5"] Oct 31 20:26:03.093351 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.093327 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-96rh5"] Oct 31 20:26:03.462460 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.462435 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:03.576065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.576035 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-openshift-ca\") pod \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " Oct 31 20:26:03.576215 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.576096 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5de424-34bd-43d9-8fd4-4bb5acd63224-kube-api-access-pvf5j\" (UniqueName: \"kubernetes.io/projected/3e5de424-34bd-43d9-8fd4-4bb5acd63224-kube-api-access-pvf5j\") pod \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " Oct 31 20:26:03.576257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.576227 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-xks-ca\") pod \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\" (UID: \"3e5de424-34bd-43d9-8fd4-4bb5acd63224\") " Oct 31 20:26:03.576388 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.576367 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-openshift-ca" pod "3e5de424-34bd-43d9-8fd4-4bb5acd63224" (UID: "3e5de424-34bd-43d9-8fd4-4bb5acd63224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:26:03.576556 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.576528 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:26:03.583160 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.583103 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-xks-ca" pod "3e5de424-34bd-43d9-8fd4-4bb5acd63224" (UID: "3e5de424-34bd-43d9-8fd4-4bb5acd63224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:26:03.583160 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.583126 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5de424-34bd-43d9-8fd4-4bb5acd63224-kube-api-access-pvf5j" pod "3e5de424-34bd-43d9-8fd4-4bb5acd63224" (UID: "3e5de424-34bd-43d9-8fd4-4bb5acd63224"). InnerVolumeSpecName "kube-api-access-pvf5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:26:03.677815 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.677780 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5de424-34bd-43d9-8fd4-4bb5acd63224-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:26:03.677815 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:03.677809 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvf5j\" (UniqueName: \"kubernetes.io/projected/3e5de424-34bd-43d9-8fd4-4bb5acd63224-kube-api-access-pvf5j\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:26:04.333295 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:04.333256 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" event={"ID":"3e5de424-34bd-43d9-8fd4-4bb5acd63224","Type":"ContainerDied","Data":"644637d391c2cfba8ed7ca617ae6d49ac80b387d260fe588df2103117050e3ab"} Oct 31 20:26:04.333567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:04.333312 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644637d391c2cfba8ed7ca617ae6d49ac80b387d260fe588df2103117050e3ab" Oct 31 20:26:04.333567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:04.333276 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m" Oct 31 20:26:04.877809 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:04.877772 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c0a218-4e58-468e-b3ec-20a51406d05b" path="/var/lib/kubelet/pods/02c0a218-4e58-468e-b3ec-20a51406d05b/volumes" Oct 31 20:26:10.263513 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:10.263482 6481 scope.go:122] "RemoveContainer" containerID="14f44f9b350d53da8663965490a02ebc867a7bfc9727edb268f722a02650f64c" Oct 31 20:26:10.272203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:10.272182 6481 scope.go:122] "RemoveContainer" containerID="1e03b7f9ef29b217ca79229fd4967984782bde84fabf158234323ebdc7f30f1e" Oct 31 20:26:10.279603 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:10.279584 6481 scope.go:122] "RemoveContainer" containerID="fdd55b6db20546f6bf9c35f12d0b32147fc7b0083ec5336163406004a8441f00" Oct 31 20:26:13.566025 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.565975 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf"] Oct 31 20:26:13.566461 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.566418 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3e5de424-34bd-43d9-8fd4-4bb5acd63224" containerName="revoke-keys" Oct 31 20:26:13.566461 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.566431 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5de424-34bd-43d9-8fd4-4bb5acd63224" containerName="revoke-keys" Oct 31 20:26:13.566549 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.566490 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5de424-34bd-43d9-8fd4-4bb5acd63224" containerName="revoke-keys" Oct 31 20:26:13.571216 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.571190 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.624743 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.624713 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d28b6b-openshift-default-dockercfg-stpdz\"" Oct 31 20:26:13.664351 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664318 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-credential-socket\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664512 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664356 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-certs\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664512 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664458 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-data\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664512 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664487 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f84e98-8263-4649-843a-f36ea219c99e-istiod-ca-cert\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664646 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664557 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-envoy\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664646 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664591 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f84e98-8263-4649-843a-f36ea219c99e-istio-podinfo\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664646 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664612 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-istio-token\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664655 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprcf\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-kube-api-access-xprcf\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.664816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.664707 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-socket\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.677686 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.677657 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf"] Oct 31 20:26:13.765633 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765592 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-envoy\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.765847 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765671 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f84e98-8263-4649-843a-f36ea219c99e-istio-podinfo\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.765847 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765711 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-istio-token\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.765847 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765756 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprcf\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-kube-api-access-xprcf\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.765847 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765800 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-socket\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.766107 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765860 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-credential-socket\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.766107 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765886 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-certs\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.766107 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.765966 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-data\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.766795 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.766763 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f84e98-8263-4649-843a-f36ea219c99e-istiod-ca-cert\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.767048 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.767022 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-credential-socket\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.767816 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.767795 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f84e98-8263-4649-843a-f36ea219c99e-istiod-ca-cert\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.768106 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.768071 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-certs\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.768333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.768315 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-data\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.774090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.769039 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-socket\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.774090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.769465 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-envoy\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.780756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.779544 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f84e98-8263-4649-843a-f36ea219c99e-istio-podinfo\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.845847 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.843678 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-istio-token\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.845847 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.844954 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprcf\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-kube-api-access-xprcf\") pod \"e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:13.884508 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:13.884471 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:14.049729 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.049695 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h"] Oct 31 20:26:14.053845 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.053821 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qhk9h"] Oct 31 20:26:14.303763 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:26:14.303733 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f84e98_8263_4649_843a_f36ea219c99e.slice/crio-bf9ba3da4a468e2dc41d1c7c141611a37a04c1b5573363c55750a763b48d955f WatchSource:0}: Error finding container bf9ba3da4a468e2dc41d1c7c141611a37a04c1b5573363c55750a763b48d955f: Status 404 returned error can't find the container with id bf9ba3da4a468e2dc41d1c7c141611a37a04c1b5573363c55750a763b48d955f Oct 31 20:26:14.305509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.305484 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf"] Oct 31 20:26:14.305973 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.305946 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:26:14.306036 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.306025 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:26:14.306072 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.306056 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:26:14.383170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.383140 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" event={"ID":"e0f84e98-8263-4649-843a-f36ea219c99e","Type":"ContainerStarted","Data":"b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b"} Oct 31 20:26:14.383316 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.383205 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" event={"ID":"e0f84e98-8263-4649-843a-f36ea219c99e","Type":"ContainerStarted","Data":"bf9ba3da4a468e2dc41d1c7c141611a37a04c1b5573363c55750a763b48d955f"} Oct 31 20:26:14.502533 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.502479 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" podStartSLOduration=1.502464741 podStartE2EDuration="1.502464741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:14.501905645 +0000 UTC m=+4145.742128774" watchObservedRunningTime="2026-10-31 20:26:14.502464741 +0000 UTC m=+4145.742687861" Oct 31 20:26:14.878738 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.878704 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c11bccc-5bdb-4084-85d0-b707fbd2ea6b" path="/var/lib/kubelet/pods/3c11bccc-5bdb-4084-85d0-b707fbd2ea6b/volumes" Oct 31 20:26:14.884837 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:14.884814 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:15.890419 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:15.890378 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:16.392877 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:16.392844 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:16.393841 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:16.393814 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:23.739379 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.739300 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7"] Oct 31 20:26:23.743041 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.743017 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.797114 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.797081 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7"] Oct 31 20:26:23.846254 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.846217 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d28b6b-serving-cert\"" Oct 31 20:26:23.846416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.846269 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 20:26:23.847621 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.847599 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xd7wl\"" Oct 31 20:26:23.872166 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.872137 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.872313 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.872290 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9976b6a0-d135-408f-8de9-20157d8e0c74-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.872437 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.872416 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvkd\" (UniqueName: \"kubernetes.io/projected/9976b6a0-d135-408f-8de9-20157d8e0c74-kube-api-access-ljvkd\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.872484 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.872467 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.973348 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.973316 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.973548 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.973416 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9976b6a0-d135-408f-8de9-20157d8e0c74-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.973548 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:26:23.973471 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-serving-cert: secret "maas-api-e2e-worker-single-main-d28b6b-serving-cert" not found Oct 31 20:26:23.973548 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:26:23.973547 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls podName:9976b6a0-d135-408f-8de9-20157d8e0c74 nodeName:}" failed. No retries permitted until 2026-10-31 20:26:24.47352587 +0000 UTC m=+4155.713748995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls") pod "maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" (UID: "9976b6a0-d135-408f-8de9-20157d8e0c74") : secret "maas-api-e2e-worker-single-main-d28b6b-serving-cert" not found Oct 31 20:26:23.973752 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.973596 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvkd\" (UniqueName: \"kubernetes.io/projected/9976b6a0-d135-408f-8de9-20157d8e0c74-kube-api-access-ljvkd\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.973752 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.973708 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.974096 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.974076 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9976b6a0-d135-408f-8de9-20157d8e0c74-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:23.976047 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:23.976024 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:24.034297 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:24.034226 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvkd\" (UniqueName: \"kubernetes.io/projected/9976b6a0-d135-408f-8de9-20157d8e0c74-kube-api-access-ljvkd\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:24.478536 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:24.478500 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:24.480882 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:24.480864 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:24.656286 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:24.656251 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:25.039146 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:26:25.039112 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9976b6a0_d135_408f_8de9_20157d8e0c74.slice/crio-c848eb13969e3ec09b304317f2bfe745691e3f02286789940090472b690b569a WatchSource:0}: Error finding container c848eb13969e3ec09b304317f2bfe745691e3f02286789940090472b690b569a: Status 404 returned error can't find the container with id c848eb13969e3ec09b304317f2bfe745691e3f02286789940090472b690b569a Oct 31 20:26:25.041654 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:25.041631 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7"] Oct 31 20:26:25.445313 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:25.445233 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" event={"ID":"9976b6a0-d135-408f-8de9-20157d8e0c74","Type":"ContainerStarted","Data":"c848eb13969e3ec09b304317f2bfe745691e3f02286789940090472b690b569a"} Oct 31 20:26:26.451820 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:26.451784 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" event={"ID":"9976b6a0-d135-408f-8de9-20157d8e0c74","Type":"ContainerStarted","Data":"d8f9313175fd7b1eb1e2396a895e5bba5bd01af36dca29787ac170c6e0c85810"} Oct 31 20:26:26.452256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:26.451849 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:26.571513 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:26.571453 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" podStartSLOduration=2.693821967 podStartE2EDuration="3.571435857s" totalImagesPullingTime="877.61389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:26.56928427 +0000 UTC m=+4157.809507399" watchObservedRunningTime="2026-10-31 20:26:26.571435857 +0000 UTC m=+4157.811658991" Oct 31 20:26:27.323770 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:26:27.323734 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:26:32.463150 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:32.463120 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:26:41.540732 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:41.540695 6481 generic.go:356] "Generic (PLEG): container finished" podID="e0f84e98-8263-4649-843a-f36ea219c99e" containerID="b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b" exitCode=0 Oct 31 20:26:41.541232 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:41.540772 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" event={"ID":"e0f84e98-8263-4649-843a-f36ea219c99e","Type":"ContainerDied","Data":"b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b"} Oct 31 20:26:41.541232 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:41.541180 6481 scope.go:122] "RemoveContainer" containerID="b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b" Oct 31 20:26:41.542534 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:41.542505 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:26:41.542600 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:41.542576 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:26:41.542636 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:41.542608 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:26:42.548054 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:42.548017 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" event={"ID":"e0f84e98-8263-4649-843a-f36ea219c99e","Type":"ContainerStarted","Data":"184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8"} Oct 31 20:26:42.895224 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:42.895126 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:42.896538 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:42.896502 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 20:26:42.896718 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:42.896564 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 20:26:43.886405 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:43.886366 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 20:26:43.887063 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:43.887032 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 20:26:44.885018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:44.884956 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 20:26:44.885242 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:44.885038 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 20:26:45.885225 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:45.885188 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 20:26:45.885853 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:45.885820 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 20:26:46.402020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:46.398127 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:46.887420 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:46.887383 6481 patch_prober.go:29] interesting pod/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 20:26:46.888080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:46.888046 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 20:26:47.889713 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:47.889682 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:47.890733 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:26:47.890711 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:26:57.323852 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:26:57.323808 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:27:10.308652 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:27:10.308622 6481 scope.go:122] "RemoveContainer" containerID="93cf2d813562273f44f9c35f38c3c76906473c60e8570f736fb8d2847354dd2b" Oct 31 20:27:10.317271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:27:10.317252 6481 scope.go:122] "RemoveContainer" containerID="94848f95df31851c23e77b6d4186eca28462968a25e5a117fe2f498a658cf4f9" Oct 31 20:27:27.324885 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:27:27.324839 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:27:57.323953 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:27:57.323867 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:28:01.848667 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:01.848633 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7"] Oct 31 20:28:01.849183 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:01.848885 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" podUID="9976b6a0-d135-408f-8de9-20157d8e0c74" containerName="maas-api" containerID="cri-o://d8f9313175fd7b1eb1e2396a895e5bba5bd01af36dca29787ac170c6e0c85810" gracePeriod=30 Oct 31 20:28:02.458013 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:02.457946 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" podUID="9976b6a0-d135-408f-8de9-20157d8e0c74" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.167:8443/health\": dial tcp 10.134.0.167:8443: connect: connection refused" Oct 31 20:28:03.015720 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.015686 6481 generic.go:356] "Generic (PLEG): container finished" podID="9976b6a0-d135-408f-8de9-20157d8e0c74" containerID="d8f9313175fd7b1eb1e2396a895e5bba5bd01af36dca29787ac170c6e0c85810" exitCode=0 Oct 31 20:28:03.016162 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.015762 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" event={"ID":"9976b6a0-d135-408f-8de9-20157d8e0c74","Type":"ContainerDied","Data":"d8f9313175fd7b1eb1e2396a895e5bba5bd01af36dca29787ac170c6e0c85810"} Oct 31 20:28:03.112401 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.112372 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:28:03.156207 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.156181 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls\") pod \"9976b6a0-d135-408f-8de9-20157d8e0c74\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " Oct 31 20:28:03.156368 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.156230 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9976b6a0-d135-408f-8de9-20157d8e0c74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9976b6a0-d135-408f-8de9-20157d8e0c74-openshift-service-ca\") pod \"9976b6a0-d135-408f-8de9-20157d8e0c74\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " Oct 31 20:28:03.156368 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.156353 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9976b6a0-d135-408f-8de9-20157d8e0c74-kube-api-access-ljvkd\" (UniqueName: \"kubernetes.io/projected/9976b6a0-d135-408f-8de9-20157d8e0c74-kube-api-access-ljvkd\") pod \"9976b6a0-d135-408f-8de9-20157d8e0c74\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " Oct 31 20:28:03.156500 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.156467 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-metrics-certs\") pod \"9976b6a0-d135-408f-8de9-20157d8e0c74\" (UID: \"9976b6a0-d135-408f-8de9-20157d8e0c74\") " Oct 31 20:28:03.156625 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.156600 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9976b6a0-d135-408f-8de9-20157d8e0c74-openshift-service-ca" pod "9976b6a0-d135-408f-8de9-20157d8e0c74" (UID: "9976b6a0-d135-408f-8de9-20157d8e0c74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:28:03.156824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.156797 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9976b6a0-d135-408f-8de9-20157d8e0c74-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:03.158476 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.158453 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls" pod "9976b6a0-d135-408f-8de9-20157d8e0c74" (UID: "9976b6a0-d135-408f-8de9-20157d8e0c74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:28:03.158571 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.158524 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9976b6a0-d135-408f-8de9-20157d8e0c74-kube-api-access-ljvkd" pod "9976b6a0-d135-408f-8de9-20157d8e0c74" (UID: "9976b6a0-d135-408f-8de9-20157d8e0c74"). InnerVolumeSpecName "kube-api-access-ljvkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:28:03.158617 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.158597 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-metrics-certs" pod "9976b6a0-d135-408f-8de9-20157d8e0c74" (UID: "9976b6a0-d135-408f-8de9-20157d8e0c74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:28:03.257700 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.257664 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:03.257700 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.257693 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9976b6a0-d135-408f-8de9-20157d8e0c74-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:03.257700 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:03.257705 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvkd\" (UniqueName: \"kubernetes.io/projected/9976b6a0-d135-408f-8de9-20157d8e0c74-kube-api-access-ljvkd\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:04.023081 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:04.023043 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" event={"ID":"9976b6a0-d135-408f-8de9-20157d8e0c74","Type":"ContainerDied","Data":"c848eb13969e3ec09b304317f2bfe745691e3f02286789940090472b690b569a"} Oct 31 20:28:04.023567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:04.023109 6481 scope.go:122] "RemoveContainer" containerID="d8f9313175fd7b1eb1e2396a895e5bba5bd01af36dca29787ac170c6e0c85810" Oct 31 20:28:04.023567 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:04.023057 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7" Oct 31 20:28:04.190794 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:04.190757 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7"] Oct 31 20:28:04.242020 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:04.241974 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d28b6b-5df5896c78-5fnd7"] Oct 31 20:28:04.878762 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:04.878731 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9976b6a0-d135-408f-8de9-20157d8e0c74" path="/var/lib/kubelet/pods/9976b6a0-d135-408f-8de9-20157d8e0c74/volumes" Oct 31 20:28:10.340038 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:10.339981 6481 scope.go:122] "RemoveContainer" containerID="3dad82935160243df390dc3c1278fd287d1021522e8e80b5acce0a7a75dc31f3" Oct 31 20:28:10.348537 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:10.348515 6481 scope.go:122] "RemoveContainer" containerID="70d6d85f909134bc0c79f7a2b23fdc5c166af9358e84c3cbe7d6a5f8e2f03364" Oct 31 20:28:10.742234 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:10.742193 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf"] Oct 31 20:28:10.742472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:10.742443 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" containerID="cri-o://184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8" gracePeriod=30 Oct 31 20:28:15.994564 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:15.994533 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:28:16.083759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.083677 6481 generic.go:356] "Generic (PLEG): container finished" podID="e0f84e98-8263-4649-843a-f36ea219c99e" containerID="184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8" exitCode=0 Oct 31 20:28:16.083759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.083726 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" Oct 31 20:28:16.083925 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.083722 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" event={"ID":"e0f84e98-8263-4649-843a-f36ea219c99e","Type":"ContainerDied","Data":"184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8"} Oct 31 20:28:16.083925 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.083839 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf" event={"ID":"e0f84e98-8263-4649-843a-f36ea219c99e","Type":"ContainerDied","Data":"bf9ba3da4a468e2dc41d1c7c141611a37a04c1b5573363c55750a763b48d955f"} Oct 31 20:28:16.083925 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.083867 6481 scope.go:122] "RemoveContainer" containerID="184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8" Oct 31 20:28:16.083925 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.083900 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-credential-socket\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.083948 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-istio-token\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-istio-token\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084033 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f84e98-8263-4649-843a-f36ea219c99e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f84e98-8263-4649-843a-f36ea219c99e-istiod-ca-cert\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084061 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-data\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084115 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-certs\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084145 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-envoy\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084206 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-socket\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084242 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0f84e98-8263-4649-843a-f36ea219c99e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f84e98-8263-4649-843a-f36ea219c99e-istio-podinfo\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084336 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-kube-api-access-xprcf\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-kube-api-access-xprcf\") pod \"e0f84e98-8263-4649-843a-f36ea219c99e\" (UID: \"e0f84e98-8263-4649-843a-f36ea219c99e\") " Oct 31 20:28:16.084472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084456 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-data" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:16.084683 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084465 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-certs" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:16.084683 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084579 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-credential-socket" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:16.084683 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084583 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-socket" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:16.084683 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084617 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f84e98-8263-4649-843a-f36ea219c99e-istiod-ca-cert" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:28:16.084906 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084734 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.084906 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084771 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.084906 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084791 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.084906 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084809 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f84e98-8263-4649-843a-f36ea219c99e-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.084906 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.084833 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.086492 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.086466 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-istio-token" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:28:16.086681 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.086622 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-kube-api-access-xprcf" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "kube-api-access-xprcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:28:16.086681 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.086657 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-envoy" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:28:16.086895 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.086872 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0f84e98-8263-4649-843a-f36ea219c99e-istio-podinfo" pod "e0f84e98-8263-4649-843a-f36ea219c99e" (UID: "e0f84e98-8263-4649-843a-f36ea219c99e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:28:16.105721 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.105698 6481 scope.go:122] "RemoveContainer" containerID="b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b" Oct 31 20:28:16.113715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.113697 6481 scope.go:122] "RemoveContainer" containerID="184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8" Oct 31 20:28:16.114002 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:28:16.113969 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8\": container with ID starting with 184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8 not found: ID does not exist" containerID="184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8" Oct 31 20:28:16.114059 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.114014 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8"} err="failed to get container status \"184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8\": rpc error: code = NotFound desc = could not find container \"184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8\": container with ID starting with 184dbab8d1aea6a16c25642a3a5df49467c86cde901787dcd71539a3491c8dd8 not found: ID does not exist" Oct 31 20:28:16.114059 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.114033 6481 scope.go:122] "RemoveContainer" containerID="b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b" Oct 31 20:28:16.114271 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:28:16.114256 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b\": container with ID starting with b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b not found: ID does not exist" containerID="b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b" Oct 31 20:28:16.114318 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.114275 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b"} err="failed to get container status \"b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b\": rpc error: code = NotFound desc = could not find container \"b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b\": container with ID starting with b4f6ce2cf9e9d41756a3fa2d63ba1a030073839be818ba66b24d18c821a10a4b not found: ID does not exist" Oct 31 20:28:16.186068 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.186035 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xprcf\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-kube-api-access-xprcf\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.186068 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.186066 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f84e98-8263-4649-843a-f36ea219c99e-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.186271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.186080 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f84e98-8263-4649-843a-f36ea219c99e-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.186271 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.186092 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f84e98-8263-4649-843a-f36ea219c99e-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:28:16.506819 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.506778 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf"] Oct 31 20:28:16.510065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.510038 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d28b6b-openshift-default-75496bb48fgvksf"] Oct 31 20:28:16.879343 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:16.879310 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" path="/var/lib/kubelet/pods/e0f84e98-8263-4649-843a-f36ea219c99e/volumes" Oct 31 20:28:27.323479 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:28:27.323441 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:28:48.097007 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:48.096966 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j"] Oct 31 20:28:48.098321 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:48.098295 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx7j"] Oct 31 20:28:48.880827 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:28:48.880793 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c40422-72db-450f-9a3d-c5faddc9cf91" path="/var/lib/kubelet/pods/38c40422-72db-450f-9a3d-c5faddc9cf91/volumes" Oct 31 20:28:57.324299 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:28:57.324263 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:29:07.006017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:07.003121 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:29:07.006017 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:07.003205 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:29:27.323714 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:29:27.323628 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:29:33.757850 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.757816 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj"] Oct 31 20:29:33.758309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758252 6481 cpu_manager.go:414] "containerMap: removing container" podUID="9976b6a0-d135-408f-8de9-20157d8e0c74" containerName="maas-api" Oct 31 20:29:33.758309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758263 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9976b6a0-d135-408f-8de9-20157d8e0c74" containerName="maas-api" Oct 31 20:29:33.758309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758275 6481 cpu_manager.go:414] "containerMap: removing container" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" Oct 31 20:29:33.758309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758280 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" Oct 31 20:29:33.758309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758298 6481 cpu_manager.go:414] "containerMap: removing container" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" Oct 31 20:29:33.758309 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758304 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" Oct 31 20:29:33.758511 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758369 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" Oct 31 20:29:33.758511 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758381 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9976b6a0-d135-408f-8de9-20157d8e0c74" containerName="maas-api" Oct 31 20:29:33.758511 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.758387 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f84e98-8263-4649-843a-f36ea219c99e" containerName="istio-proxy" Oct 31 20:29:33.761575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.761558 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.814629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.814589 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj"] Oct 31 20:29:33.863956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.863914 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:29:33.864170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.863987 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:29:33.893975 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.893941 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.894180 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.894039 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvb8\" (UniqueName: \"kubernetes.io/projected/3b18bcf4-196d-4ca8-be83-fae07cd72680-kube-api-access-7hvb8\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.894180 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.894086 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.995096 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.995057 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.995325 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.995131 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvb8\" (UniqueName: \"kubernetes.io/projected/3b18bcf4-196d-4ca8-be83-fae07cd72680-kube-api-access-7hvb8\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.995325 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.995187 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.996014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.995937 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:33.997427 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:33.997407 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:34.053883 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:34.053804 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvb8\" (UniqueName: \"kubernetes.io/projected/3b18bcf4-196d-4ca8-be83-fae07cd72680-kube-api-access-7hvb8\") pod \"maas-api-revoke-sub-452b6d7b-qbgmj\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:34.074676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:34.074643 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:34.442140 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:29:34.442105 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b18bcf4_196d_4ca8_be83_fae07cd72680.slice/crio-e58c0cf71d03fb05dd34338b1f4f42d53c118c5e288cdfe9e8235209e7355d8e WatchSource:0}: Error finding container e58c0cf71d03fb05dd34338b1f4f42d53c118c5e288cdfe9e8235209e7355d8e: Status 404 returned error can't find the container with id e58c0cf71d03fb05dd34338b1f4f42d53c118c5e288cdfe9e8235209e7355d8e Oct 31 20:29:34.443945 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:34.443918 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj"] Oct 31 20:29:34.572177 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:34.572145 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" event={"ID":"3b18bcf4-196d-4ca8-be83-fae07cd72680","Type":"ContainerStarted","Data":"e58c0cf71d03fb05dd34338b1f4f42d53c118c5e288cdfe9e8235209e7355d8e"} Oct 31 20:29:35.583209 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:35.583113 6481 generic.go:356] "Generic (PLEG): container finished" podID="3b18bcf4-196d-4ca8-be83-fae07cd72680" containerID="bc0f430c81c7217512a92204fd7fb0bb5c5c624d34026d855b1bf27034844df8" exitCode=0 Oct 31 20:29:35.583209 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:35.583177 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" event={"ID":"3b18bcf4-196d-4ca8-be83-fae07cd72680","Type":"ContainerDied","Data":"bc0f430c81c7217512a92204fd7fb0bb5c5c624d34026d855b1bf27034844df8"} Oct 31 20:29:37.390014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.386217 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:37.398023 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.397671 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b18bcf4-196d-4ca8-be83-fae07cd72680-kube-api-access-7hvb8\" (UniqueName: \"kubernetes.io/projected/3b18bcf4-196d-4ca8-be83-fae07cd72680-kube-api-access-7hvb8\") pod \"3b18bcf4-196d-4ca8-be83-fae07cd72680\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " Oct 31 20:29:37.398023 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.397845 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-openshift-ca\") pod \"3b18bcf4-196d-4ca8-be83-fae07cd72680\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " Oct 31 20:29:37.398023 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.397925 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-xks-ca\") pod \"3b18bcf4-196d-4ca8-be83-fae07cd72680\" (UID: \"3b18bcf4-196d-4ca8-be83-fae07cd72680\") " Oct 31 20:29:37.406012 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.403208 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-openshift-ca" pod "3b18bcf4-196d-4ca8-be83-fae07cd72680" (UID: "3b18bcf4-196d-4ca8-be83-fae07cd72680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:29:37.406475 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.406449 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-xks-ca" pod "3b18bcf4-196d-4ca8-be83-fae07cd72680" (UID: "3b18bcf4-196d-4ca8-be83-fae07cd72680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:29:37.415813 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.415786 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b18bcf4-196d-4ca8-be83-fae07cd72680-kube-api-access-7hvb8" pod "3b18bcf4-196d-4ca8-be83-fae07cd72680" (UID: "3b18bcf4-196d-4ca8-be83-fae07cd72680"). InnerVolumeSpecName "kube-api-access-7hvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:29:37.499046 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.499010 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:29:37.499319 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.499297 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hvb8\" (UniqueName: \"kubernetes.io/projected/3b18bcf4-196d-4ca8-be83-fae07cd72680-kube-api-access-7hvb8\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:29:37.499457 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.499443 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18bcf4-196d-4ca8-be83-fae07cd72680-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:29:37.676074 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.673946 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" event={"ID":"3b18bcf4-196d-4ca8-be83-fae07cd72680","Type":"ContainerDied","Data":"e58c0cf71d03fb05dd34338b1f4f42d53c118c5e288cdfe9e8235209e7355d8e"} Oct 31 20:29:37.680016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.678088 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e58c0cf71d03fb05dd34338b1f4f42d53c118c5e288cdfe9e8235209e7355d8e" Oct 31 20:29:37.680016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:37.678178 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj" Oct 31 20:29:57.256897 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.256863 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch"] Oct 31 20:29:57.257339 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.257325 6481 cpu_manager.go:414] "containerMap: removing container" podUID="3b18bcf4-196d-4ca8-be83-fae07cd72680" containerName="revoke-keys" Oct 31 20:29:57.257385 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.257341 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b18bcf4-196d-4ca8-be83-fae07cd72680" containerName="revoke-keys" Oct 31 20:29:57.257424 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.257410 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b18bcf4-196d-4ca8-be83-fae07cd72680" containerName="revoke-keys" Oct 31 20:29:57.260463 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.260447 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.313903 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.313864 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch"] Oct 31 20:29:57.328891 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:29:57.328855 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:29:57.362725 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.362689 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:29:57.362898 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.362754 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:29:57.418305 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.418150 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklsq\" (UniqueName: \"kubernetes.io/projected/6617784f-61fa-466a-9ffe-7f65f73bdc1b-kube-api-access-mklsq\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.418509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.418323 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.418509 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.418439 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.519911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.519835 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.519911 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.519874 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.520122 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.520057 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklsq\" (UniqueName: \"kubernetes.io/projected/6617784f-61fa-466a-9ffe-7f65f73bdc1b-kube-api-access-mklsq\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.520516 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.520495 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.522315 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.522290 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.578148 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.578115 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklsq\" (UniqueName: \"kubernetes.io/projected/6617784f-61fa-466a-9ffe-7f65f73bdc1b-kube-api-access-mklsq\") pod \"maas-api-revoke-sub-3e31a506-98kch\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:57.873485 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:57.873393 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:29:58.240381 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:29:58.240351 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6617784f_61fa_466a_9ffe_7f65f73bdc1b.slice/crio-6aecc1a75b619cad68a686f8d58802aac10c8f2c472cf7adfa17ef78fb8e222a WatchSource:0}: Error finding container 6aecc1a75b619cad68a686f8d58802aac10c8f2c472cf7adfa17ef78fb8e222a: Status 404 returned error can't find the container with id 6aecc1a75b619cad68a686f8d58802aac10c8f2c472cf7adfa17ef78fb8e222a Oct 31 20:29:58.241759 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:58.241734 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch"] Oct 31 20:29:58.798948 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:58.798897 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" event={"ID":"6617784f-61fa-466a-9ffe-7f65f73bdc1b","Type":"ContainerStarted","Data":"95b861b16ee274a46d1b8bf9e81c58c0482f93f8a43ef7a5b0efe2cb3e6b61b8"} Oct 31 20:29:58.799486 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:58.798962 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" event={"ID":"6617784f-61fa-466a-9ffe-7f65f73bdc1b","Type":"ContainerStarted","Data":"6aecc1a75b619cad68a686f8d58802aac10c8f2c472cf7adfa17ef78fb8e222a"} Oct 31 20:29:58.916551 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:58.916477 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" podStartSLOduration=1.916458214 podStartE2EDuration="1.916458214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:29:58.91391804 +0000 UTC m=+4370.154141170" watchObservedRunningTime="2026-10-31 20:29:58.916458214 +0000 UTC m=+4370.156681344" Oct 31 20:29:59.817916 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:59.817813 6481 generic.go:356] "Generic (PLEG): container finished" podID="6617784f-61fa-466a-9ffe-7f65f73bdc1b" containerID="95b861b16ee274a46d1b8bf9e81c58c0482f93f8a43ef7a5b0efe2cb3e6b61b8" exitCode=0 Oct 31 20:29:59.818639 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:29:59.818343 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" event={"ID":"6617784f-61fa-466a-9ffe-7f65f73bdc1b","Type":"ContainerDied","Data":"95b861b16ee274a46d1b8bf9e81c58c0482f93f8a43ef7a5b0efe2cb3e6b61b8"} Oct 31 20:30:00.562016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:00.561222 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-rxs7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:30:00.562016 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:00.561306 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rxs7f" podUID="fe5137bc-2028-4ac9-bc02-354efba8d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:30:00.785014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:00.784854 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mdfhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:30:00.785014 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:00.784936 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mdfhv" podUID="afb1bd7d-63fa-43ed-8674-9299055d81c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:30:01.137671 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.137635 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:30:01.303143 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.303110 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6617784f-61fa-466a-9ffe-7f65f73bdc1b-kube-api-access-mklsq\" (UniqueName: \"kubernetes.io/projected/6617784f-61fa-466a-9ffe-7f65f73bdc1b-kube-api-access-mklsq\") pod \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " Oct 31 20:30:01.303320 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.303175 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-openshift-ca\") pod \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " Oct 31 20:30:01.303365 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.303336 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-xks-ca\") pod \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\" (UID: \"6617784f-61fa-466a-9ffe-7f65f73bdc1b\") " Oct 31 20:30:01.303551 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.303526 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-openshift-ca" pod "6617784f-61fa-466a-9ffe-7f65f73bdc1b" (UID: "6617784f-61fa-466a-9ffe-7f65f73bdc1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:30:01.303711 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.303693 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:01.305349 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.305317 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6617784f-61fa-466a-9ffe-7f65f73bdc1b-kube-api-access-mklsq" pod "6617784f-61fa-466a-9ffe-7f65f73bdc1b" (UID: "6617784f-61fa-466a-9ffe-7f65f73bdc1b"). InnerVolumeSpecName "kube-api-access-mklsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:30:01.305349 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.305325 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-xks-ca" pod "6617784f-61fa-466a-9ffe-7f65f73bdc1b" (UID: "6617784f-61fa-466a-9ffe-7f65f73bdc1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:01.405334 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.405243 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mklsq\" (UniqueName: \"kubernetes.io/projected/6617784f-61fa-466a-9ffe-7f65f73bdc1b-kube-api-access-mklsq\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:01.405334 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.405276 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6617784f-61fa-466a-9ffe-7f65f73bdc1b-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:01.835671 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.835640 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" Oct 31 20:30:01.835671 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.835654 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch" event={"ID":"6617784f-61fa-466a-9ffe-7f65f73bdc1b","Type":"ContainerDied","Data":"6aecc1a75b619cad68a686f8d58802aac10c8f2c472cf7adfa17ef78fb8e222a"} Oct 31 20:30:01.835941 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:01.835697 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aecc1a75b619cad68a686f8d58802aac10c8f2c472cf7adfa17ef78fb8e222a" Oct 31 20:30:10.388645 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:10.388610 6481 scope.go:122] "RemoveContainer" containerID="73578ecbc85353a3fa9a4427135baf7c4cc3c6f423deb19b27bf4e3750a839af" Oct 31 20:30:23.452226 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.452187 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52"] Oct 31 20:30:23.452641 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.452627 6481 cpu_manager.go:414] "containerMap: removing container" podUID="6617784f-61fa-466a-9ffe-7f65f73bdc1b" containerName="revoke-keys" Oct 31 20:30:23.452687 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.452641 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6617784f-61fa-466a-9ffe-7f65f73bdc1b" containerName="revoke-keys" Oct 31 20:30:23.452730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.452718 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6617784f-61fa-466a-9ffe-7f65f73bdc1b" containerName="revoke-keys" Oct 31 20:30:23.455863 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.455847 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.509452 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.509416 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52"] Oct 31 20:30:23.532800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.532763 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.532800 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.532803 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.533077 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.533022 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp6h\" (UniqueName: \"kubernetes.io/projected/60624b6a-d384-4239-9667-bb79c81bc309-kube-api-access-qnp6h\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.558055 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.558031 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:30:23.558422 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.558406 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ctqxt\"" Oct 31 20:30:23.633872 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.633838 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.634090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.633949 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp6h\" (UniqueName: \"kubernetes.io/projected/60624b6a-d384-4239-9667-bb79c81bc309-kube-api-access-qnp6h\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.634090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.633980 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.634551 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.634523 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.636217 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.636198 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.692398 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.692363 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp6h\" (UniqueName: \"kubernetes.io/projected/60624b6a-d384-4239-9667-bb79c81bc309-kube-api-access-qnp6h\") pod \"maas-api-revoke-sub-d4b429d8-vlx52\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:23.768558 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:23.768527 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:24.135675 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:30:24.135207 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60624b6a_d384_4239_9667_bb79c81bc309.slice/crio-0eb95a1cfae51db8175d79e0a381f7200b0e7c4eda3b90f90bc03f00cf578709 WatchSource:0}: Error finding container 0eb95a1cfae51db8175d79e0a381f7200b0e7c4eda3b90f90bc03f00cf578709: Status 404 returned error can't find the container with id 0eb95a1cfae51db8175d79e0a381f7200b0e7c4eda3b90f90bc03f00cf578709 Oct 31 20:30:24.137090 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:24.137062 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52"] Oct 31 20:30:25.015883 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:25.015847 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" event={"ID":"60624b6a-d384-4239-9667-bb79c81bc309","Type":"ContainerStarted","Data":"cbda825ff0ec7eae2be6626044dd515b6ba2b9372d690ec863fd28620e09690d"} Oct 31 20:30:25.016316 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:25.015903 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" event={"ID":"60624b6a-d384-4239-9667-bb79c81bc309","Type":"ContainerStarted","Data":"0eb95a1cfae51db8175d79e0a381f7200b0e7c4eda3b90f90bc03f00cf578709"} Oct 31 20:30:25.132805 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:25.132752 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" podStartSLOduration=2.132738489 podStartE2EDuration="2.132738489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:25.131820912 +0000 UTC m=+4396.372044041" watchObservedRunningTime="2026-10-31 20:30:25.132738489 +0000 UTC m=+4396.372961618" Oct 31 20:30:26.022480 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:26.022442 6481 generic.go:356] "Generic (PLEG): container finished" podID="60624b6a-d384-4239-9667-bb79c81bc309" containerID="cbda825ff0ec7eae2be6626044dd515b6ba2b9372d690ec863fd28620e09690d" exitCode=0 Oct 31 20:30:26.022950 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:26.022521 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" event={"ID":"60624b6a-d384-4239-9667-bb79c81bc309","Type":"ContainerDied","Data":"cbda825ff0ec7eae2be6626044dd515b6ba2b9372d690ec863fd28620e09690d"} Oct 31 20:30:27.162281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.162253 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:27.266256 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.266226 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-xks-ca\") pod \"60624b6a-d384-4239-9667-bb79c81bc309\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " Oct 31 20:30:27.266416 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.266278 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60624b6a-d384-4239-9667-bb79c81bc309-kube-api-access-qnp6h\" (UniqueName: \"kubernetes.io/projected/60624b6a-d384-4239-9667-bb79c81bc309-kube-api-access-qnp6h\") pod \"60624b6a-d384-4239-9667-bb79c81bc309\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " Oct 31 20:30:27.266465 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.266405 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-openshift-ca\") pod \"60624b6a-d384-4239-9667-bb79c81bc309\" (UID: \"60624b6a-d384-4239-9667-bb79c81bc309\") " Oct 31 20:30:27.266730 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.266702 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-openshift-ca" pod "60624b6a-d384-4239-9667-bb79c81bc309" (UID: "60624b6a-d384-4239-9667-bb79c81bc309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:30:27.266850 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.266797 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-openshift-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:27.268394 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.268371 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60624b6a-d384-4239-9667-bb79c81bc309-kube-api-access-qnp6h" pod "60624b6a-d384-4239-9667-bb79c81bc309" (UID: "60624b6a-d384-4239-9667-bb79c81bc309"). InnerVolumeSpecName "kube-api-access-qnp6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:30:27.268481 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.268371 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-xks-ca" pod "60624b6a-d384-4239-9667-bb79c81bc309" (UID: "60624b6a-d384-4239-9667-bb79c81bc309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:27.324706 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:30:27.324626 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:30:27.367487 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.367459 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnp6h\" (UniqueName: \"kubernetes.io/projected/60624b6a-d384-4239-9667-bb79c81bc309-kube-api-access-qnp6h\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:27.367660 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:27.367491 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60624b6a-d384-4239-9667-bb79c81bc309-maas-api-xks-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:28.032954 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:28.032925 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" Oct 31 20:30:28.032954 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:28.032935 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52" event={"ID":"60624b6a-d384-4239-9667-bb79c81bc309","Type":"ContainerDied","Data":"0eb95a1cfae51db8175d79e0a381f7200b0e7c4eda3b90f90bc03f00cf578709"} Oct 31 20:30:28.033231 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:28.033016 6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb95a1cfae51db8175d79e0a381f7200b0e7c4eda3b90f90bc03f00cf578709" Oct 31 20:30:36.045254 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.045211 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv"] Oct 31 20:30:36.097580 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.097538 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qpxvv"] Oct 31 20:30:36.568874 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.568834 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9"] Oct 31 20:30:36.569715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.569686 6481 cpu_manager.go:414] "containerMap: removing container" podUID="60624b6a-d384-4239-9667-bb79c81bc309" containerName="revoke-keys" Oct 31 20:30:36.569715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.569715 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60624b6a-d384-4239-9667-bb79c81bc309" containerName="revoke-keys" Oct 31 20:30:36.569919 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.569828 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="60624b6a-d384-4239-9667-bb79c81bc309" containerName="revoke-keys" Oct 31 20:30:36.573527 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.573502 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.624424 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.624385 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9"] Oct 31 20:30:36.666985 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.666951 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-openshift-service-ca\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.667156 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.667099 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-maas-api-tls\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.667251 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.667229 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-metrics-certs\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.667316 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.667301 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-kube-api-access-f7wg2\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.677984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.677960 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 20:30:36.677984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.677973 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xd7wl\"" Oct 31 20:30:36.677984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.677970 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 20:30:36.767926 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.767893 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-metrics-certs\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.768157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.767939 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-kube-api-access-f7wg2\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.768157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.767974 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-openshift-service-ca\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.768157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.768061 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-maas-api-tls\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.768824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.768797 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-openshift-service-ca\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.770535 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.770516 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-metrics-certs\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.770634 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.770554 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-maas-api-tls\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.826375 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.826304 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-kube-api-access-f7wg2\") pod \"maas-api-7d5f466c97-vwcf9\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:36.878203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.878171 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7da11ba-8ba2-44e0-b2b5-567aadae0fd1" path="/var/lib/kubelet/pods/f7da11ba-8ba2-44e0-b2b5-567aadae0fd1/volumes" Oct 31 20:30:36.887180 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:36.887151 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:37.254426 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:30:37.254394 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f750a35_9ba2_4a76_beec_7a9b7eadfaf7.slice/crio-ba9109edc44f1ee6ba9c0a41689300cc13f87642976bf9195b53e0f18342ffd0 WatchSource:0}: Error finding container ba9109edc44f1ee6ba9c0a41689300cc13f87642976bf9195b53e0f18342ffd0: Status 404 returned error can't find the container with id ba9109edc44f1ee6ba9c0a41689300cc13f87642976bf9195b53e0f18342ffd0 Oct 31 20:30:37.255789 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:37.255767 6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:30:37.256695 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:37.256673 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9"] Oct 31 20:30:38.080699 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:38.080669 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" event={"ID":"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7","Type":"ContainerStarted","Data":"ba9109edc44f1ee6ba9c0a41689300cc13f87642976bf9195b53e0f18342ffd0"} Oct 31 20:30:39.087454 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:39.087413 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" event={"ID":"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7","Type":"ContainerStarted","Data":"a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1"} Oct 31 20:30:39.087862 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:39.087483 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:39.202825 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:39.202773 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" podStartSLOduration=2.338784888 podStartE2EDuration="3.202758915s" totalImagesPullingTime="863.974027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:39.201347108 +0000 UTC m=+4410.441570236" watchObservedRunningTime="2026-10-31 20:30:39.202758915 +0000 UTC m=+4410.442982043" Oct 31 20:30:43.468875 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.468840 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw"] Oct 31 20:30:43.472657 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.472637 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.549919 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.549881 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f056d87f-35bd-4594-93ca-ff6679dff741-istiod-ca-cert\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.549932 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-istio-token\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.550038 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-credential-socket\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550145 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.550086 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f056d87f-35bd-4594-93ca-ff6679dff741-istio-podinfo\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550281 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.550243 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-certs\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.550295 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kk6\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-kube-api-access-49kk6\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550402 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.550380 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-data\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550469 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.550425 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-envoy\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.550530 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.550464 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-socket\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.574690 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.574648 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw"] Oct 31 20:30:43.626166 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.625302 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-47884b3f-openshift-default-dockercfg-597wr\"" Oct 31 20:30:43.651101 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651063 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-istio-token\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651114 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-credential-socket\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651156 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f056d87f-35bd-4594-93ca-ff6679dff741-istio-podinfo\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651249 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-certs\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651333 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651316 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kk6\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-kube-api-access-49kk6\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651345 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-data\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651387 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-envoy\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651423 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-socket\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651480 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f056d87f-35bd-4594-93ca-ff6679dff741-istiod-ca-cert\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651773 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651612 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-credential-socket\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651877 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651851 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-data\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651948 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651885 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-socket\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.651948 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.651933 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-certs\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.652259 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.652238 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f056d87f-35bd-4594-93ca-ff6679dff741-istiod-ca-cert\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.653787 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.653750 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-envoy\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.653984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.653963 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f056d87f-35bd-4594-93ca-ff6679dff741-istio-podinfo\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.723219 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.716477 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-istio-token\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.723219 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.719780 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kk6\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-kube-api-access-49kk6\") pod \"e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:43.800342 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:43.800308 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:44.213854 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:30:44.213820 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf056d87f_35bd_4594_93ca_ff6679dff741.slice/crio-997e83e5e60f4c65ed8fbe36024e2cd9d575fffa706b331811ad7f5cc3a15ae0 WatchSource:0}: Error finding container 997e83e5e60f4c65ed8fbe36024e2cd9d575fffa706b331811ad7f5cc3a15ae0: Status 404 returned error can't find the container with id 997e83e5e60f4c65ed8fbe36024e2cd9d575fffa706b331811ad7f5cc3a15ae0 Oct 31 20:30:44.216294 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:44.216256 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:30:44.216392 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:44.216340 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:30:44.216392 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:44.216383 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:30:44.217147 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:44.217121 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw"] Oct 31 20:30:45.046754 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.046724 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9"] Oct 31 20:30:45.047231 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.046931 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" podUID="5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" containerName="maas-api" containerID="cri-o://a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1" gracePeriod=30 Oct 31 20:30:45.053751 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.053731 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:45.078312 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.078285 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz"] Oct 31 20:30:45.081956 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.081936 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.118496 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.118468 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" event={"ID":"f056d87f-35bd-4594-93ca-ff6679dff741","Type":"ContainerStarted","Data":"e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7"} Oct 31 20:30:45.118613 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.118524 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" event={"ID":"f056d87f-35bd-4594-93ca-ff6679dff741","Type":"ContainerStarted","Data":"997e83e5e60f4c65ed8fbe36024e2cd9d575fffa706b331811ad7f5cc3a15ae0"} Oct 31 20:30:45.165229 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.165192 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz"] Oct 31 20:30:45.165897 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.165864 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1976a259-154d-4a45-975d-623af74462f9-metrics-certs\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.166080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.165938 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x46\" (UniqueName: \"kubernetes.io/projected/1976a259-154d-4a45-975d-623af74462f9-kube-api-access-f7x46\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.166080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.165971 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1976a259-154d-4a45-975d-623af74462f9-openshift-service-ca\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.166080 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.166043 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1976a259-154d-4a45-975d-623af74462f9-maas-api-tls\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.267079 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.267047 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x46\" (UniqueName: \"kubernetes.io/projected/1976a259-154d-4a45-975d-623af74462f9-kube-api-access-f7x46\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.267249 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.267082 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1976a259-154d-4a45-975d-623af74462f9-openshift-service-ca\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.267249 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.267112 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1976a259-154d-4a45-975d-623af74462f9-maas-api-tls\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.267249 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.267236 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1976a259-154d-4a45-975d-623af74462f9-metrics-certs\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.267863 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.267825 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1976a259-154d-4a45-975d-623af74462f9-openshift-service-ca\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.269675 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.269650 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1976a259-154d-4a45-975d-623af74462f9-maas-api-tls\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.269790 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.269764 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1976a259-154d-4a45-975d-623af74462f9-metrics-certs\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.288126 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.288085 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podStartSLOduration=2.288071013 podStartE2EDuration="2.288071013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:45.287252608 +0000 UTC m=+4416.527475738" watchObservedRunningTime="2026-10-31 20:30:45.288071013 +0000 UTC m=+4416.528294142" Oct 31 20:30:45.304030 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.303960 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:45.325350 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.325323 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x46\" (UniqueName: \"kubernetes.io/projected/1976a259-154d-4a45-975d-623af74462f9-kube-api-access-f7x46\") pod \"maas-api-74f5c976df-qz7xz\" (UID: \"1976a259-154d-4a45-975d-623af74462f9\") " pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.367568 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.367541 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-kube-api-access-f7wg2\") pod \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " Oct 31 20:30:45.367735 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.367601 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-metrics-certs\") pod \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " Oct 31 20:30:45.367735 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.367625 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-openshift-service-ca\") pod \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " Oct 31 20:30:45.367735 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.367662 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-maas-api-tls\") pod \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\" (UID: \"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7\") " Oct 31 20:30:45.368089 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.368054 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-openshift-service-ca" pod "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" (UID: "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:30:45.369804 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.369775 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-metrics-certs" pod "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" (UID: "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:45.369900 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.369820 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-maas-api-tls" pod "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" (UID: "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:30:45.369900 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.369840 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-kube-api-access-f7wg2" pod "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" (UID: "5f750a35-9ba2-4a76-beec-7a9b7eadfaf7"). InnerVolumeSpecName "kube-api-access-f7wg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:30:45.434844 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.434808 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:45.469380 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.469331 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7wg2\" (UniqueName: \"kubernetes.io/projected/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-kube-api-access-f7wg2\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:45.469380 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.469372 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:45.469380 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.469383 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:45.469380 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.469392 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:30:45.800586 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.800554 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:45.800858 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:30:45.800821 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1976a259_154d_4a45_975d_623af74462f9.slice/crio-4022722aac1e15e1625d279875076302a49bb0433f47c70a8534617809f7402c WatchSource:0}: Error finding container 4022722aac1e15e1625d279875076302a49bb0433f47c70a8534617809f7402c: Status 404 returned error can't find the container with id 4022722aac1e15e1625d279875076302a49bb0433f47c70a8534617809f7402c Oct 31 20:30:45.804864 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.804835 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz"] Oct 31 20:30:45.806209 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:45.806186 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:46.124672 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.124633 6481 generic.go:356] "Generic (PLEG): container finished" podID="5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" containerID="a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1" exitCode=0 Oct 31 20:30:46.125158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.124679 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" Oct 31 20:30:46.125158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.124725 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" event={"ID":"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7","Type":"ContainerDied","Data":"a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1"} Oct 31 20:30:46.125158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.124777 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9" event={"ID":"5f750a35-9ba2-4a76-beec-7a9b7eadfaf7","Type":"ContainerDied","Data":"ba9109edc44f1ee6ba9c0a41689300cc13f87642976bf9195b53e0f18342ffd0"} Oct 31 20:30:46.125158 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.124800 6481 scope.go:122] "RemoveContainer" containerID="a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1" Oct 31 20:30:46.126269 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.126246 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" event={"ID":"1976a259-154d-4a45-975d-623af74462f9","Type":"ContainerStarted","Data":"4022722aac1e15e1625d279875076302a49bb0433f47c70a8534617809f7402c"} Oct 31 20:30:46.126621 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.126601 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:46.127877 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.127857 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:30:46.137197 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.137165 6481 scope.go:122] "RemoveContainer" containerID="a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1" Oct 31 20:30:46.137725 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:30:46.137695 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1\": container with ID starting with a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1 not found: ID does not exist" containerID="a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1" Oct 31 20:30:46.137808 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.137732 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1"} err="failed to get container status \"a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1\": rpc error: code = NotFound desc = could not find container \"a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1\": container with ID starting with a1c8572f847746aa305c39607f5593482d23d903ca734347fc0af2c0c6ca96d1 not found: ID does not exist" Oct 31 20:30:46.411047 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.410944 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9"] Oct 31 20:30:46.413933 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.413914 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d5f466c97-vwcf9"] Oct 31 20:30:46.878417 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:46.878371 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" path="/var/lib/kubelet/pods/5f750a35-9ba2-4a76-beec-7a9b7eadfaf7/volumes" Oct 31 20:30:47.134532 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:47.134484 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" event={"ID":"1976a259-154d-4a45-975d-623af74462f9","Type":"ContainerStarted","Data":"915f6da62ed00425068ef36a949de12b84228b3c0c0d6d8cb2a47f20cf31d787"} Oct 31 20:30:47.135160 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:47.134557 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:47.257746 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:47.257690 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" podStartSLOduration=1.401009675 podStartE2EDuration="2.257675147s" totalImagesPullingTime="856.665472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:47.255629128 +0000 UTC m=+4418.495852257" watchObservedRunningTime="2026-10-31 20:30:47.257675147 +0000 UTC m=+4418.497898276" Oct 31 20:30:51.925729 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:51.925648 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6"] Oct 31 20:30:51.926164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:51.926070 6481 cpu_manager.go:414] "containerMap: removing container" podUID="5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" containerName="maas-api" Oct 31 20:30:51.926164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:51.926084 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" containerName="maas-api" Oct 31 20:30:51.926241 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:51.926166 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f750a35-9ba2-4a76-beec-7a9b7eadfaf7" containerName="maas-api" Oct 31 20:30:51.929457 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:51.929437 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:51.985942 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:51.985911 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-47884b3f-serving-cert\"" Oct 31 20:30:51.993361 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:51.993331 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6"] Oct 31 20:30:52.037422 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.037383 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400071b8-84fe-4924-a029-4b4288e8e9c9-openshift-service-ca\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.037611 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.037470 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-metrics-certs\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.037611 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.037528 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9d5\" (UniqueName: \"kubernetes.io/projected/400071b8-84fe-4924-a029-4b4288e8e9c9-kube-api-access-gv9d5\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.037751 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.037666 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.138887 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.138847 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400071b8-84fe-4924-a029-4b4288e8e9c9-openshift-service-ca\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.139076 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.138914 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-metrics-certs\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.139076 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.139015 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9d5\" (UniqueName: \"kubernetes.io/projected/400071b8-84fe-4924-a029-4b4288e8e9c9-kube-api-access-gv9d5\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.139263 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.139090 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.139263 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:30:52.139196 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-serving-cert: secret "maas-api-e2e-ait-47884b3f-serving-cert" not found Oct 31 20:30:52.139263 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:30:52.139246 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls podName:400071b8-84fe-4924-a029-4b4288e8e9c9 nodeName:}" failed. No retries permitted until 2026-10-31 20:30:52.639231705 +0000 UTC m=+4423.879454825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls") pod "maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" (UID: "400071b8-84fe-4924-a029-4b4288e8e9c9") : secret "maas-api-e2e-ait-47884b3f-serving-cert" not found Oct 31 20:30:52.139532 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.139511 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400071b8-84fe-4924-a029-4b4288e8e9c9-openshift-service-ca\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.141369 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.141352 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-metrics-certs\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.212634 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.207830 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9d5\" (UniqueName: \"kubernetes.io/projected/400071b8-84fe-4924-a029-4b4288e8e9c9-kube-api-access-gv9d5\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.646786 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.646690 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.649097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.649074 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls\") pod \"maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:52.842824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:52.842793 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:53.147787 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:53.147756 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74f5c976df-qz7xz" Oct 31 20:30:53.214721 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:30:53.214689 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400071b8_84fe_4924_a029_4b4288e8e9c9.slice/crio-92567e8c9524947541361ee03d8b5a004f790c1b3f3d653553d65a561bdd97d9 WatchSource:0}: Error finding container 92567e8c9524947541361ee03d8b5a004f790c1b3f3d653553d65a561bdd97d9: Status 404 returned error can't find the container with id 92567e8c9524947541361ee03d8b5a004f790c1b3f3d653553d65a561bdd97d9 Oct 31 20:30:53.256140 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:53.256113 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6"] Oct 31 20:30:54.171195 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:54.171156 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" event={"ID":"400071b8-84fe-4924-a029-4b4288e8e9c9","Type":"ContainerStarted","Data":"405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1"} Oct 31 20:30:54.171640 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:54.171219 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" event={"ID":"400071b8-84fe-4924-a029-4b4288e8e9c9","Type":"ContainerStarted","Data":"92567e8c9524947541361ee03d8b5a004f790c1b3f3d653553d65a561bdd97d9"} Oct 31 20:30:54.171640 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:54.171264 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:30:54.288788 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:30:54.288734 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" podStartSLOduration=2.419741551 podStartE2EDuration="3.288721311s" totalImagesPullingTime="868.97976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:54.287130773 +0000 UTC m=+4425.527353903" watchObservedRunningTime="2026-10-31 20:30:54.288721311 +0000 UTC m=+4425.528944436" Oct 31 20:30:57.324730 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:30:57.324694 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:31:00.183738 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:00.183710 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:31:04.048230 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:04.048198 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m"] Oct 31 20:31:04.099609 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:04.099570 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-tm89m"] Oct 31 20:31:04.892284 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:04.892246 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5de424-34bd-43d9-8fd4-4bb5acd63224" path="/var/lib/kubelet/pods/3e5de424-34bd-43d9-8fd4-4bb5acd63224/volumes" Oct 31 20:31:10.258467 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:10.258430 6481 generic.go:356] "Generic (PLEG): container finished" podID="f056d87f-35bd-4594-93ca-ff6679dff741" containerID="e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7" exitCode=0 Oct 31 20:31:10.259052 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:10.258512 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" event={"ID":"f056d87f-35bd-4594-93ca-ff6679dff741","Type":"ContainerDied","Data":"e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7"} Oct 31 20:31:10.259052 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:10.259037 6481 scope.go:122] "RemoveContainer" containerID="e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7" Oct 31 20:31:10.260251 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:10.260225 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:31:10.260319 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:10.260292 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:31:10.260357 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:10.260323 6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 20:31:11.278019 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:11.264981 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" event={"ID":"f056d87f-35bd-4594-93ca-ff6679dff741","Type":"ContainerStarted","Data":"7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35"} Oct 31 20:31:11.800539 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:11.800472 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:31:11.801593 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:11.801567 6481 patch_prober.go:29] interesting pod/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 20:31:11.801721 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:11.801614 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 20:31:12.801131 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:12.801093 6481 patch_prober.go:29] interesting pod/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 20:31:12.801531 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:12.801148 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 20:31:13.800626 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:13.800586 6481 patch_prober.go:29] interesting pod/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 20:31:13.800824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:13.800656 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 20:31:14.801766 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:14.801722 6481 patch_prober.go:29] interesting pod/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 20:31:14.802205 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:14.801789 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 20:31:15.801320 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:15.801283 6481 patch_prober.go:29] interesting pod/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 20:31:15.801512 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:15.801358 6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 20:31:16.127276 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:16.127177 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:31:16.805132 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:16.805101 6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:31:16.806061 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:16.806039 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:31:27.324709 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:31:27.324666 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:31:41.700329 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:41.700291 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6"] Oct 31 20:31:41.701341 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:41.700657 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" podUID="400071b8-84fe-4924-a029-4b4288e8e9c9" containerName="maas-api" containerID="cri-o://405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1" gracePeriod=30 Oct 31 20:31:41.964719 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:41.964692 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:31:42.099609 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.099288 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400071b8-84fe-4924-a029-4b4288e8e9c9-kube-api-access-gv9d5\" (UniqueName: \"kubernetes.io/projected/400071b8-84fe-4924-a029-4b4288e8e9c9-kube-api-access-gv9d5\") pod \"400071b8-84fe-4924-a029-4b4288e8e9c9\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " Oct 31 20:31:42.099609 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.099348 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-metrics-certs\") pod \"400071b8-84fe-4924-a029-4b4288e8e9c9\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " Oct 31 20:31:42.100021 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.099973 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls\") pod \"400071b8-84fe-4924-a029-4b4288e8e9c9\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " Oct 31 20:31:42.100163 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.100072 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400071b8-84fe-4924-a029-4b4288e8e9c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400071b8-84fe-4924-a029-4b4288e8e9c9-openshift-service-ca\") pod \"400071b8-84fe-4924-a029-4b4288e8e9c9\" (UID: \"400071b8-84fe-4924-a029-4b4288e8e9c9\") " Oct 31 20:31:42.100849 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.100815 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400071b8-84fe-4924-a029-4b4288e8e9c9-openshift-service-ca" pod "400071b8-84fe-4924-a029-4b4288e8e9c9" (UID: "400071b8-84fe-4924-a029-4b4288e8e9c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:31:42.106164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.106135 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-metrics-certs" pod "400071b8-84fe-4924-a029-4b4288e8e9c9" (UID: "400071b8-84fe-4924-a029-4b4288e8e9c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:31:42.109094 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.109066 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400071b8-84fe-4924-a029-4b4288e8e9c9-kube-api-access-gv9d5" pod "400071b8-84fe-4924-a029-4b4288e8e9c9" (UID: "400071b8-84fe-4924-a029-4b4288e8e9c9"). InnerVolumeSpecName "kube-api-access-gv9d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:31:42.115025 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.114962 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls" pod "400071b8-84fe-4924-a029-4b4288e8e9c9" (UID: "400071b8-84fe-4924-a029-4b4288e8e9c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:31:42.201777 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.201686 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9d5\" (UniqueName: \"kubernetes.io/projected/400071b8-84fe-4924-a029-4b4288e8e9c9-kube-api-access-gv9d5\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:42.201777 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.201722 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:42.201777 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.201735 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400071b8-84fe-4924-a029-4b4288e8e9c9-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:42.201777 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.201749 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400071b8-84fe-4924-a029-4b4288e8e9c9-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:42.450479 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.450392 6481 generic.go:356] "Generic (PLEG): container finished" podID="400071b8-84fe-4924-a029-4b4288e8e9c9" containerID="405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1" exitCode=0 Oct 31 20:31:42.450479 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.450454 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" Oct 31 20:31:42.450672 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.450470 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" event={"ID":"400071b8-84fe-4924-a029-4b4288e8e9c9","Type":"ContainerDied","Data":"405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1"} Oct 31 20:31:42.450672 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.450548 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6" event={"ID":"400071b8-84fe-4924-a029-4b4288e8e9c9","Type":"ContainerDied","Data":"92567e8c9524947541361ee03d8b5a004f790c1b3f3d653553d65a561bdd97d9"} Oct 31 20:31:42.450672 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.450570 6481 scope.go:122] "RemoveContainer" containerID="405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1" Oct 31 20:31:42.460257 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.460236 6481 scope.go:122] "RemoveContainer" containerID="405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1" Oct 31 20:31:42.460502 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:31:42.460482 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1\": container with ID starting with 405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1 not found: ID does not exist" containerID="405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1" Oct 31 20:31:42.460557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.460511 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1"} err="failed to get container status \"405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1\": rpc error: code = NotFound desc = could not find container \"405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1\": container with ID starting with 405fa24adc8d47b90bbd319448d1e7c1020bfa9619a2f891366df167a288e5f1 not found: ID does not exist" Oct 31 20:31:42.617741 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.617711 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6"] Oct 31 20:31:42.621018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.620978 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47884b3f-7778f76c55-bd2j6"] Oct 31 20:31:42.878037 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:42.877981 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400071b8-84fe-4924-a029-4b4288e8e9c9" path="/var/lib/kubelet/pods/400071b8-84fe-4924-a029-4b4288e8e9c9/volumes" Oct 31 20:31:44.666550 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:44.666513 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw"] Oct 31 20:31:44.667120 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:44.666732 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" containerID="cri-o://7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35" gracePeriod=30 Oct 31 20:31:46.808194 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:46.808157 6481 patch_prober.go:29] interesting pod/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:31:46.808617 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:46.808239 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:31:49.916517 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:49.916484 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:31:50.087821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.087784 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-socket\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.087836 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-kube-api-access-49kk6\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-kube-api-access-49kk6\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.087898 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f056d87f-35bd-4594-93ca-ff6679dff741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f056d87f-35bd-4594-93ca-ff6679dff741-istiod-ca-cert\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.087940 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-istio-token\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-istio-token\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088056 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088042 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-certs\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088094 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-credential-socket\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088136 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-envoy\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088183 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f056d87f-35bd-4594-93ca-ff6679dff741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f056d87f-35bd-4594-93ca-ff6679dff741-istio-podinfo\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088168 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-socket" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:31:50.088299 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088237 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-data\") pod \"f056d87f-35bd-4594-93ca-ff6679dff741\" (UID: \"f056d87f-35bd-4594-93ca-ff6679dff741\") " Oct 31 20:31:50.088557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088396 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-credential-socket" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:31:50.088557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088432 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f056d87f-35bd-4594-93ca-ff6679dff741-istiod-ca-cert" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:31:50.088557 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088519 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-certs" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:31:50.088728 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088632 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-data" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:31:50.088728 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088721 6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-data\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.088840 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088738 6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.088840 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088755 6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f056d87f-35bd-4594-93ca-ff6679dff741-istiod-ca-cert\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.088840 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088771 6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-workload-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.088840 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.088785 6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-credential-socket\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.091472 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.091444 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-kube-api-access-49kk6" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "kube-api-access-49kk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:31:50.091612 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.091475 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-envoy" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:31:50.094468 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.094441 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f056d87f-35bd-4594-93ca-ff6679dff741-istio-podinfo" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:31:50.094571 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.094496 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-istio-token" pod "f056d87f-35bd-4594-93ca-ff6679dff741" (UID: "f056d87f-35bd-4594-93ca-ff6679dff741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:31:50.189969 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.189928 6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f056d87f-35bd-4594-93ca-ff6679dff741-istio-envoy\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.189969 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.189962 6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f056d87f-35bd-4594-93ca-ff6679dff741-istio-podinfo\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.189969 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.189973 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kk6\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-kube-api-access-49kk6\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.189969 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.189983 6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f056d87f-35bd-4594-93ca-ff6679dff741-istio-token\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:31:50.503577 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.503540 6481 generic.go:356] "Generic (PLEG): container finished" podID="f056d87f-35bd-4594-93ca-ff6679dff741" containerID="7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35" exitCode=0 Oct 31 20:31:50.503773 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.503593 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" Oct 31 20:31:50.503773 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.503622 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" event={"ID":"f056d87f-35bd-4594-93ca-ff6679dff741","Type":"ContainerDied","Data":"7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35"} Oct 31 20:31:50.503773 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.503682 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw" event={"ID":"f056d87f-35bd-4594-93ca-ff6679dff741","Type":"ContainerDied","Data":"997e83e5e60f4c65ed8fbe36024e2cd9d575fffa706b331811ad7f5cc3a15ae0"} Oct 31 20:31:50.503773 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.503711 6481 scope.go:122] "RemoveContainer" containerID="7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35" Oct 31 20:31:50.513691 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.513666 6481 scope.go:122] "RemoveContainer" containerID="e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7" Oct 31 20:31:50.522164 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.522140 6481 scope.go:122] "RemoveContainer" containerID="7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35" Oct 31 20:31:50.522456 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:31:50.522438 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35\": container with ID starting with 7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35 not found: ID does not exist" containerID="7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35" Oct 31 20:31:50.522528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.522465 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35"} err="failed to get container status \"7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35\": rpc error: code = NotFound desc = could not find container \"7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35\": container with ID starting with 7b92a9b657d5a71b4ea53df12dca4dc56fede549c53c6c0abd36dcbb9a2b3f35 not found: ID does not exist" Oct 31 20:31:50.522528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.522482 6481 scope.go:122] "RemoveContainer" containerID="e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7" Oct 31 20:31:50.522747 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:31:50.522729 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7\": container with ID starting with e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7 not found: ID does not exist" containerID="e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7" Oct 31 20:31:50.522795 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.522755 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7"} err="failed to get container status \"e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7\": rpc error: code = NotFound desc = could not find container \"e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7\": container with ID starting with e6aaf0797487e8dd8422eea2475db046d57183abde34a66200cc9852b04fb0c7 not found: ID does not exist" Oct 31 20:31:50.691447 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.691407 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw"] Oct 31 20:31:50.742912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.742880 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-47884b3f-openshift-default-5dd95d7ccc-slbrw"] Oct 31 20:31:50.878647 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:50.878560 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" path="/var/lib/kubelet/pods/f056d87f-35bd-4594-93ca-ff6679dff741/volumes" Oct 31 20:31:53.647711 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.647674 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz"] Oct 31 20:31:53.648231 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.648213 6481 cpu_manager.go:414] "containerMap: removing container" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" Oct 31 20:31:53.648282 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.648233 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" Oct 31 20:31:53.648282 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.648261 6481 cpu_manager.go:414] "containerMap: removing container" podUID="400071b8-84fe-4924-a029-4b4288e8e9c9" containerName="maas-api" Oct 31 20:31:53.648282 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.648266 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400071b8-84fe-4924-a029-4b4288e8e9c9" containerName="maas-api" Oct 31 20:31:53.648377 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.648350 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" Oct 31 20:31:53.648377 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.648366 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" Oct 31 20:31:53.648448 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.648379 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="400071b8-84fe-4924-a029-4b4288e8e9c9" containerName="maas-api" Oct 31 20:31:53.653044 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.653016 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.708123 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.708094 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-12dd5c-serving-cert\"" Oct 31 20:31:53.761370 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.761335 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz"] Oct 31 20:31:53.826304 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.826264 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.826514 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.826401 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd48f\" (UniqueName: \"kubernetes.io/projected/16396c5e-c9e4-4994-b6bf-bc064575c9c3-kube-api-access-vd48f\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.826514 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.826480 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.826514 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.826507 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16396c5e-c9e4-4994-b6bf-bc064575c9c3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.927938 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.927849 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.928383 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.928349 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16396c5e-c9e4-4994-b6bf-bc064575c9c3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.928508 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.928447 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.928575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.928559 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd48f\" (UniqueName: \"kubernetes.io/projected/16396c5e-c9e4-4994-b6bf-bc064575c9c3-kube-api-access-vd48f\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.928932 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:31:53.928289 6481 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-serving-cert: secret "maas-api-e2e-authcfg-http-12dd5c-serving-cert" not found Oct 31 20:31:53.929078 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:31:53.928979 6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls podName:16396c5e-c9e4-4994-b6bf-bc064575c9c3 nodeName:}" failed. No retries permitted until 2026-10-31 20:31:54.428957452 +0000 UTC m=+4485.669180575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls") pod "maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" (UID: "16396c5e-c9e4-4994-b6bf-bc064575c9c3") : secret "maas-api-e2e-authcfg-http-12dd5c-serving-cert" not found Oct 31 20:31:53.929775 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.929751 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16396c5e-c9e4-4994-b6bf-bc064575c9c3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.948729 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.948701 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:53.986594 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:53.986565 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd48f\" (UniqueName: \"kubernetes.io/projected/16396c5e-c9e4-4994-b6bf-bc064575c9c3-kube-api-access-vd48f\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:54.434711 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:54.434675 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:54.437098 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:54.437072 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:54.566791 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:54.566758 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:54.938276 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:31:54.938249 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16396c5e_c9e4_4994_b6bf_bc064575c9c3.slice/crio-ffb610600f3b2bf4b94a925faa5b35bf4c8722f2cff54bd0ef39086a41db06d3 WatchSource:0}: Error finding container ffb610600f3b2bf4b94a925faa5b35bf4c8722f2cff54bd0ef39086a41db06d3: Status 404 returned error can't find the container with id ffb610600f3b2bf4b94a925faa5b35bf4c8722f2cff54bd0ef39086a41db06d3 Oct 31 20:31:54.940756 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:54.940679 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz"] Oct 31 20:31:55.538085 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:55.538043 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" event={"ID":"16396c5e-c9e4-4994-b6bf-bc064575c9c3","Type":"ContainerStarted","Data":"ffb610600f3b2bf4b94a925faa5b35bf4c8722f2cff54bd0ef39086a41db06d3"} Oct 31 20:31:56.545709 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:56.545673 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" event={"ID":"16396c5e-c9e4-4994-b6bf-bc064575c9c3","Type":"ContainerStarted","Data":"5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4"} Oct 31 20:31:56.546229 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:56.545795 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:31:56.667385 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:31:56.667339 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" podStartSLOduration=2.699547832 podStartE2EDuration="3.667325142s" totalImagesPullingTime="967.77731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:56.665400661 +0000 UTC m=+4487.905623790" watchObservedRunningTime="2026-10-31 20:31:56.667325142 +0000 UTC m=+4487.907548272" Oct 31 20:31:57.325271 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:31:57.325234 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:32:02.556646 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:32:02.556609 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:32:10.424116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:32:10.424082 6481 scope.go:122] "RemoveContainer" containerID="ac11aa7723e3c0d52bd806fbd7a2ddce45c3eb6dcf54e85e2cffe081dd08094f" Oct 31 20:32:10.433081 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:32:10.432961 6481 scope.go:122] "RemoveContainer" containerID="adb1307891c31ec9482b3050c973031ba57977e61f8fbab04f591f21e3da402a" Oct 31 20:32:27.323920 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:32:27.323837 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:32:57.323793 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:32:57.323755 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:33:27.325075 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:33:27.325033 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:33:56.480857 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.480594 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz"] Oct 31 20:33:56.482065 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.482011 6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" podUID="16396c5e-c9e4-4994-b6bf-bc064575c9c3" containerName="maas-api" containerID="cri-o://5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4" gracePeriod=30 Oct 31 20:33:56.751988 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.751963 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:33:56.881118 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.881046 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16396c5e-c9e4-4994-b6bf-bc064575c9c3-kube-api-access-vd48f\" (UniqueName: \"kubernetes.io/projected/16396c5e-c9e4-4994-b6bf-bc064575c9c3-kube-api-access-vd48f\") pod \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " Oct 31 20:33:56.881312 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.881122 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16396c5e-c9e4-4994-b6bf-bc064575c9c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16396c5e-c9e4-4994-b6bf-bc064575c9c3-openshift-service-ca\") pod \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " Oct 31 20:33:56.881312 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.881191 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-metrics-certs\") pod \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " Oct 31 20:33:56.881312 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.881266 6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls\") pod \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\" (UID: \"16396c5e-c9e4-4994-b6bf-bc064575c9c3\") " Oct 31 20:33:56.882575 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.882547 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16396c5e-c9e4-4994-b6bf-bc064575c9c3-openshift-service-ca" pod "16396c5e-c9e4-4994-b6bf-bc064575c9c3" (UID: "16396c5e-c9e4-4994-b6bf-bc064575c9c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:33:56.885739 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.885699 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls" pod "16396c5e-c9e4-4994-b6bf-bc064575c9c3" (UID: "16396c5e-c9e4-4994-b6bf-bc064575c9c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:33:56.885968 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.885946 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16396c5e-c9e4-4994-b6bf-bc064575c9c3-kube-api-access-vd48f" pod "16396c5e-c9e4-4994-b6bf-bc064575c9c3" (UID: "16396c5e-c9e4-4994-b6bf-bc064575c9c3"). InnerVolumeSpecName "kube-api-access-vd48f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:33:56.887558 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.887529 6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-metrics-certs" pod "16396c5e-c9e4-4994-b6bf-bc064575c9c3" (UID: "16396c5e-c9e4-4994-b6bf-bc064575c9c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:33:56.990578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.990421 6481 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-metrics-certs\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:56.990578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.990464 6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16396c5e-c9e4-4994-b6bf-bc064575c9c3-maas-api-tls\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:56.990578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.990483 6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd48f\" (UniqueName: \"kubernetes.io/projected/16396c5e-c9e4-4994-b6bf-bc064575c9c3-kube-api-access-vd48f\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:56.990578 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:56.990500 6481 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16396c5e-c9e4-4994-b6bf-bc064575c9c3-openshift-service-ca\") on node \"ip-10-0-143-134.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 20:33:57.125491 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.125455 6481 generic.go:356] "Generic (PLEG): container finished" podID="16396c5e-c9e4-4994-b6bf-bc064575c9c3" containerID="5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4" exitCode=0 Oct 31 20:33:57.125666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.125498 6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" Oct 31 20:33:57.125666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.125541 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" event={"ID":"16396c5e-c9e4-4994-b6bf-bc064575c9c3","Type":"ContainerDied","Data":"5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4"} Oct 31 20:33:57.125666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.125596 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz" event={"ID":"16396c5e-c9e4-4994-b6bf-bc064575c9c3","Type":"ContainerDied","Data":"ffb610600f3b2bf4b94a925faa5b35bf4c8722f2cff54bd0ef39086a41db06d3"} Oct 31 20:33:57.125666 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.125620 6481 scope.go:122] "RemoveContainer" containerID="5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4" Oct 31 20:33:57.134528 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.134510 6481 scope.go:122] "RemoveContainer" containerID="5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4" Oct 31 20:33:57.134803 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:33:57.134779 6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4\": container with ID starting with 5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4 not found: ID does not exist" containerID="5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4" Oct 31 20:33:57.134853 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.134812 6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4"} err="failed to get container status \"5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4\": rpc error: code = NotFound desc = could not find container \"5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4\": container with ID starting with 5e371566e57dfa90196011e5038519a672f81d551ffd14a81c90e62965a222a4 not found: ID does not exist" Oct 31 20:33:57.292391 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.292297 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz"] Oct 31 20:33:57.324956 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:33:57.324920 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:33:57.343298 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:57.343272 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-12dd5c-588bb5468c-vndqz"] Oct 31 20:33:58.880078 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:33:58.880044 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16396c5e-c9e4-4994-b6bf-bc064575c9c3" path="/var/lib/kubelet/pods/16396c5e-c9e4-4994-b6bf-bc064575c9c3/volumes" Oct 31 20:34:27.371252 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:34:27.371201 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:34:38.054887 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:34:38.054848 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj"] Oct 31 20:34:38.105926 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:34:38.105893 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qbgmj"] Oct 31 20:34:38.879235 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:34:38.879198 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b18bcf4-196d-4ca8-be83-fae07cd72680" path="/var/lib/kubelet/pods/3b18bcf4-196d-4ca8-be83-fae07cd72680/volumes" Oct 31 20:34:52.003018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:34:52.002119 6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-6qfjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:34:52.003018 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:34:52.002191 6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-6qfjx" podUID="14094e68-424c-4b92-b951-506da45efa7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:34:57.324171 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:34:57.324133 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:35:01.054432 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:01.054398 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch"] Oct 31 20:35:01.105301 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:01.105265 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-98kch"] Oct 31 20:35:02.878460 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:02.878418 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6617784f-61fa-466a-9ffe-7f65f73bdc1b" path="/var/lib/kubelet/pods/6617784f-61fa-466a-9ffe-7f65f73bdc1b/volumes" Oct 31 20:35:27.324799 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:35:27.324720 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:35:27.972807 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:27.972775 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b5mbb_6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2/manager/0.log" Oct 31 20:35:28.048559 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:28.048525 6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52"] Oct 31 20:35:28.099582 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:28.099539 6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b429d8-vlx52"] Oct 31 20:35:28.879337 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:28.879300 6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60624b6a-d384-4239-9667-bb79c81bc309" path="/var/lib/kubelet/pods/60624b6a-d384-4239-9667-bb79c81bc309/volumes" Oct 31 20:35:29.004948 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:29.004909 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69947f5d84-8lqlr_72efb8b4-3301-404e-b4d1-a43275992fd8/manager/0.log" Oct 31 20:35:29.469097 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:29.469070 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mtl9j_ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f/server/0.log" Oct 31 20:35:30.009629 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:30.009598 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x49nm_dd44a116-ca3d-4836-b52d-ed8fe300b085/proxy/0.log" Oct 31 20:35:30.999125 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:30.999099 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74f5c976df-qz7xz_1976a259-154d-4a45-975d-623af74462f9/maas-api/0.log" Oct 31 20:35:31.881563 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:31.881535 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-szzc5_60fc2851-39a2-4652-bec6-53e5ecc8f4fb/postgres/0.log" Oct 31 20:35:34.221831 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:34.221797 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pw67w_b93b628d-f933-44de-a0b9-42301b2524cc/kuadrant-console-plugin/0.log" Oct 31 20:35:36.226649 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:36.226608 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vjzx5_31be8330-7a6e-46fa-aa22-da8cf0c90c3c/discovery/0.log" Oct 31 20:35:36.711965 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:36.711941 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b655b5cf-ndjvz_ea81bab8-4e72-4032-83bd-b559eab23229/kube-auth-proxy/0.log" Oct 31 20:35:36.917917 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:36.917886 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-6qfjx_14094e68-424c-4b92-b951-506da45efa7e/istio-proxy/0.log" Oct 31 20:35:37.409306 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:37.409273 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-f9fls_20b9145e-b5bb-4776-a30f-a904b9e01370/istio-proxy/0.log" Oct 31 20:35:37.887437 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:37.887408 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-mdfhv_afb1bd7d-63fa-43ed-8674-9299055d81c9/istio-proxy/0.log" Oct 31 20:35:38.331781 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:38.331751 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-rxs7f_fe5137bc-2028-4ac9-bc02-354efba8d848/istio-proxy/0.log" Oct 31 20:35:39.436984 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:39.436956 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-5jvp6_f9095efd-a3e2-4a0d-b9cd-1a380eef1d39/payload-pre-processing/0.log" Oct 31 20:35:39.657572 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:39.657537 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ff9cw_3522c641-6153-4239-90e0-c7a55a55bafb/payload-processing/0.log" Oct 31 20:35:40.769502 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:40.769474 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q272n_6f8e9398-998f-42be-8dee-f0e169426697/main/0.log" Oct 31 20:35:41.234855 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:41.234824 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-hcnz5_038fbde1-ebd7-45b2-a2e2-acd0c92e1870/main/0.log" Oct 31 20:35:41.662255 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:41.662183 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2_9de1e1d1-192a-488e-8f42-e013800a9dcf/main/0.log" Oct 31 20:35:42.101453 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:42.101421 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn_7e7c0d83-8f3b-4130-9469-1d70e37eb9b3/main/0.log" Oct 31 20:35:52.618336 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:52.618304 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b5mbb_6ea7565c-19d2-4bd9-b37a-9cb2b15dd4c2/manager/0.log" Oct 31 20:35:53.459795 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:53.459763 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69947f5d84-8lqlr_72efb8b4-3301-404e-b4d1-a43275992fd8/manager/0.log" Oct 31 20:35:53.821846 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:53.821772 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mtl9j_ba68fe55-84a9-4ae2-b46e-9eb0bc4ab71f/server/0.log" Oct 31 20:35:54.365170 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:54.365143 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x49nm_dd44a116-ca3d-4836-b52d-ed8fe300b085/proxy/0.log" Oct 31 20:35:55.180624 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:55.180589 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74f5c976df-qz7xz_1976a259-154d-4a45-975d-623af74462f9/maas-api/0.log" Oct 31 20:35:56.069892 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:56.069860 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-szzc5_60fc2851-39a2-4652-bec6-53e5ecc8f4fb/postgres/0.log" Oct 31 20:35:57.324509 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:35:57.324474 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:35:58.024157 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:58.024127 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pw67w_b93b628d-f933-44de-a0b9-42301b2524cc/kuadrant-console-plugin/0.log" Oct 31 20:35:59.715262 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:35:59.715228 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vjzx5_31be8330-7a6e-46fa-aa22-da8cf0c90c3c/discovery/0.log" Oct 31 20:36:00.236076 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:00.236036 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b655b5cf-ndjvz_ea81bab8-4e72-4032-83bd-b559eab23229/kube-auth-proxy/0.log" Oct 31 20:36:00.448377 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:00.448348 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-6qfjx_14094e68-424c-4b92-b951-506da45efa7e/istio-proxy/0.log" Oct 31 20:36:00.932713 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:00.932685 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-f9fls_20b9145e-b5bb-4776-a30f-a904b9e01370/istio-proxy/0.log" Oct 31 20:36:01.396627 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:01.396595 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-mdfhv_afb1bd7d-63fa-43ed-8674-9299055d81c9/istio-proxy/0.log" Oct 31 20:36:01.810926 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:01.810895 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-rxs7f_fe5137bc-2028-4ac9-bc02-354efba8d848/istio-proxy/0.log" Oct 31 20:36:03.007747 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:03.007710 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-5jvp6_f9095efd-a3e2-4a0d-b9cd-1a380eef1d39/payload-pre-processing/0.log" Oct 31 20:36:03.220362 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:03.220334 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ff9cw_3522c641-6153-4239-90e0-c7a55a55bafb/payload-processing/0.log" Oct 31 20:36:04.053676 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:04.053646 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q272n_6f8e9398-998f-42be-8dee-f0e169426697/main/0.log" Oct 31 20:36:04.544163 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:04.544133 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-hcnz5_038fbde1-ebd7-45b2-a2e2-acd0c92e1870/main/0.log" Oct 31 20:36:04.979154 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:04.979125 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86m5m2_9de1e1d1-192a-488e-8f42-e013800a9dcf/main/0.log" Oct 31 20:36:05.413031 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:05.412946 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6qldn_7e7c0d83-8f3b-4130-9469-1d70e37eb9b3/main/0.log" Oct 31 20:36:10.494752 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:10.494719 6481 scope.go:122] "RemoveContainer" containerID="95b861b16ee274a46d1b8bf9e81c58c0482f93f8a43ef7a5b0efe2cb3e6b61b8" Oct 31 20:36:10.503378 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:10.503356 6481 scope.go:122] "RemoveContainer" containerID="bc0f430c81c7217512a92204fd7fb0bb5c5c624d34026d855b1bf27034844df8" Oct 31 20:36:27.323817 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:36:27.323778 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:36:33.575824 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:33.575791 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69947f5d84-8lqlr_72efb8b4-3301-404e-b4d1-a43275992fd8/manager/0.log" Oct 31 20:36:34.067821 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:34.067784 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69947f5d84-8lqlr_72efb8b4-3301-404e-b4d1-a43275992fd8/manager/0.log" Oct 31 20:36:34.855685 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:34.855651 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ff9cw_3522c641-6153-4239-90e0-c7a55a55bafb/payload-processing/0.log" Oct 31 20:36:35.079674 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:36:35.079646 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ff9cw_3522c641-6153-4239-90e0-c7a55a55bafb/payload-processing/0.log" Oct 31 20:36:57.324309 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:36:57.324228 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:37:10.525879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:10.525846 6481 scope.go:122] "RemoveContainer" containerID="cbda825ff0ec7eae2be6626044dd515b6ba2b9372d690ec863fd28620e09690d" Oct 31 20:37:11.984134 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:11.984098 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69947f5d84-8lqlr_72efb8b4-3301-404e-b4d1-a43275992fd8/manager/0.log" Oct 31 20:37:12.531376 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:12.531343 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69947f5d84-8lqlr_72efb8b4-3301-404e-b4d1-a43275992fd8/manager/0.log" Oct 31 20:37:13.303086 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:13.303060 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ff9cw_3522c641-6153-4239-90e0-c7a55a55bafb/payload-processing/0.log" Oct 31 20:37:13.527939 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:13.527913 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ff9cw_3522c641-6153-4239-90e0-c7a55a55bafb/payload-processing/0.log" Oct 31 20:37:24.069534 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:24.069500 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gttls_fb981019-7d15-475e-ac02-ef33f915867a/global-pull-secret-syncer/0.log" Oct 31 20:37:26.898393 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:26.898363 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mxn7z_9976ed46-0a87-46f9-982d-4a4ac667e834/konnectivity-agent/0.log" Oct 31 20:37:27.324703 ip-10-0-143-134 kubenswrapper[6481]: E1031 20:37:27.324667 6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dx4jp" containerName="extractor" Oct 31 20:37:28.499669 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:28.499640 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-134.us-west-2.compute.internal_76ead912f9e2b59f12d28159dae3cb89/haproxy/0.log" Oct 31 20:37:32.708083 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.708049 6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl"] Oct 31 20:37:32.708663 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.708648 6481 cpu_manager.go:414] "containerMap: removing container" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" Oct 31 20:37:32.708715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.708664 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f056d87f-35bd-4594-93ca-ff6679dff741" containerName="istio-proxy" Oct 31 20:37:32.708715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.708693 6481 cpu_manager.go:414] "containerMap: removing container" podUID="16396c5e-c9e4-4994-b6bf-bc064575c9c3" containerName="maas-api" Oct 31 20:37:32.708715 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.708700 6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16396c5e-c9e4-4994-b6bf-bc064575c9c3" containerName="maas-api" Oct 31 20:37:32.708855 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.708806 6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="16396c5e-c9e4-4994-b6bf-bc064575c9c3" containerName="maas-api" Oct 31 20:37:32.711912 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.711894 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.749814 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.749783 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-lib-modules\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.749814 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.749815 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-proc\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.749981 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.749835 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-sys\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.749981 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.749856 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-podres\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.750100 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.750047 6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqx2p\" (UniqueName: \"kubernetes.io/projected/c62b491d-8eee-4336-a502-95496a8c35ae-kube-api-access-tqx2p\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.765038 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.765014 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ks7ld\"/\"openshift-service-ca.crt\"" Oct 31 20:37:32.765133 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.765040 6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ks7ld\"/\"kube-root-ca.crt\"" Oct 31 20:37:32.816375 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.816349 6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ks7ld\"/\"default-dockercfg-lbbt2\"" Oct 31 20:37:32.818273 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.818250 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl"] Oct 31 20:37:32.850921 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.850890 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-lib-modules\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.850926 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-proc\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.850956 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-sys\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.851018 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-podres\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.851046 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-proc\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.851055 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-sys\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.851046 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-lib-modules\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851116 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.851116 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c62b491d-8eee-4336-a502-95496a8c35ae-podres\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.851478 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.851180 6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqx2p\" (UniqueName: \"kubernetes.io/projected/c62b491d-8eee-4336-a502-95496a8c35ae-kube-api-access-tqx2p\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:32.909290 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:32.909262 6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqx2p\" (UniqueName: \"kubernetes.io/projected/c62b491d-8eee-4336-a502-95496a8c35ae-kube-api-access-tqx2p\") pod \"perf-node-gather-daemonset-cgpvl\" (UID: \"c62b491d-8eee-4336-a502-95496a8c35ae\") " pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:33.019241 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:33.019220 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pw67w_b93b628d-f933-44de-a0b9-42301b2524cc/kuadrant-console-plugin/0.log" Oct 31 20:37:33.024409 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:33.024389 6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:33.386849 ip-10-0-143-134 kubenswrapper[6481]: W1031 20:37:33.386777 6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc62b491d_8eee_4336_a502_95496a8c35ae.slice/crio-24a0b40cc80669df645b2b02d241a89104ce0b7748054ee9151c5871e8cbb378 WatchSource:0}: Error finding container 24a0b40cc80669df645b2b02d241a89104ce0b7748054ee9151c5871e8cbb378: Status 404 returned error can't find the container with id 24a0b40cc80669df645b2b02d241a89104ce0b7748054ee9151c5871e8cbb378 Oct 31 20:37:33.388879 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:33.388852 6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl"] Oct 31 20:37:34.352104 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:34.352063 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" event={"ID":"c62b491d-8eee-4336-a502-95496a8c35ae","Type":"ContainerStarted","Data":"02837fa8f8c80712b08c7b2f40c022c29cf4bababbefb5e6191855e02362fd6e"} Oct 31 20:37:34.352550 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:34.352132 6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" event={"ID":"c62b491d-8eee-4336-a502-95496a8c35ae","Type":"ContainerStarted","Data":"24a0b40cc80669df645b2b02d241a89104ce0b7748054ee9151c5871e8cbb378"} Oct 31 20:37:34.352550 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:34.352162 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:34.468626 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:34.468574 6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" podStartSLOduration=2.468560846 podStartE2EDuration="2.468560846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:37:34.467316204 +0000 UTC m=+4825.707539332" watchObservedRunningTime="2026-10-31 20:37:34.468560846 +0000 UTC m=+4825.708783974" Oct 31 20:37:40.366978 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:40.366948 6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ks7ld/perf-node-gather-daemonset-cgpvl" Oct 31 20:37:41.028199 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:41.028166 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75bb7ff6dd-6qm4t_df7cc259-0fb7-4f12-ac63-1b798ada6303/metrics-server/0.log" Oct 31 20:37:41.415010 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:41.414922 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-628gq_06e5e0e1-8e0c-4224-bdea-f2a70ef90dcc/monitoring-plugin/0.log" Oct 31 20:37:42.967006 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:42.966960 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqmbj_73474282-1f87-4204-969d-2ddcbb5ee758/node-exporter/0.log" Oct 31 20:37:43.350203 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:43.350174 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqmbj_73474282-1f87-4204-969d-2ddcbb5ee758/kube-rbac-proxy/0.log" Oct 31 20:37:43.733638 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:43.733609 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqmbj_73474282-1f87-4204-969d-2ddcbb5ee758/init-textfile/0.log" Oct 31 20:37:49.197541 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:49.197508 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4j7np_8e993f09-bc04-46f4-9c3a-c77ba0341b81/prometheus-operator/0.log" Oct 31 20:37:49.578142 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:49.578114 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4j7np_8e993f09-bc04-46f4-9c3a-c77ba0341b81/kube-rbac-proxy/0.log" Oct 31 20:37:50.353646 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:50.353619 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69454877bc-frmmb_edfc0931-1a29-4aaf-b2e2-954b25a473e5/telemeter-client/0.log" Oct 31 20:37:50.739332 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:50.739307 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69454877bc-frmmb_edfc0931-1a29-4aaf-b2e2-954b25a473e5/reload/0.log" Oct 31 20:37:51.123233 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:51.123162 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69454877bc-frmmb_edfc0931-1a29-4aaf-b2e2-954b25a473e5/kube-rbac-proxy/0.log" Oct 31 20:37:51.515785 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:51.515758 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d45575d-5ns87_264ad1b2-15f9-4b71-97ef-96b7f380c650/thanos-query/0.log" Oct 31 20:37:51.901981 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:51.901901 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d45575d-5ns87_264ad1b2-15f9-4b71-97ef-96b7f380c650/kube-rbac-proxy-web/0.log" Oct 31 20:37:52.288175 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:52.288121 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d45575d-5ns87_264ad1b2-15f9-4b71-97ef-96b7f380c650/kube-rbac-proxy/0.log" Oct 31 20:37:52.672362 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:52.672277 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d45575d-5ns87_264ad1b2-15f9-4b71-97ef-96b7f380c650/prom-label-proxy/0.log" Oct 31 20:37:53.057308 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:53.057271 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d45575d-5ns87_264ad1b2-15f9-4b71-97ef-96b7f380c650/kube-rbac-proxy-rules/0.log" Oct 31 20:37:53.442424 ip-10-0-143-134 kubenswrapper[6481]: I1031 20:37:53.442341 6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d45575d-5ns87_264ad1b2-15f9-4b71-97ef-96b7f380c650/kube-rbac-proxy-metrics/0.log"